Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide functionality to set Pervasive's stderr to unbuffered bahaviour (Unix-like behaviour) #6973

Closed
vicuna opened this issue Aug 28, 2015 · 13 comments

Comments

@vicuna
Copy link

vicuna commented Aug 28, 2015

Original bug ID: 6973
Reporter: oliver
Status: acknowledged (set by @damiendoligez on 2015-11-27T15:02:27Z)
Resolution: open
Priority: normal
Severity: feature
Platform: ./.
OS: ./.
OS Version: ./.
Version: 4.02.3
Category: standard library
Has duplicate: #7177
Monitored by: @diml @hcarty

Bug description

stderr on Unix is unbuffered by default.

In OCaml, all the Channels from Pervasives are buffered by default,
and flush-function must be used.

This issue is counter-intuitive (error must be reported ASAP) and un-Unix-ish, even though OCaml is typically used in Unix environment.

It would be fine if there would be a possibility to set
stderr-behaviour from Pervasives to behave like Unix-default behaviour
of stderr.

Something like an aequivalent of setbuf(3) / setvbuf(3) for OCaml's pervasives would be nice.
(Either at least for Pervasive's stderr, or in general for streams, like setbuf/setvbuf, which are for all streams.)

See W.R. Stevens, APUE, 14th printing 1997, Chapter 5.4.:

"The standard error stream for example, is normally unbuffered." (page 123)

@vicuna
Copy link
Author

vicuna commented Sep 1, 2015

Comment author: Str

(as has been said on caml-list)

the _endline and _newline print functions are doing a flush on the output channel

see stdlib/pervasives.ml:419

@github-actions
Copy link

This issue has been open one year with no activity. Consequently, it is being marked with the "stale" label. What this means is that the issue will be automatically closed in 30 days unless more comments are added or the "stale" label is removed. Comments that provide new information on the issue are especially welcome: is it still reproducible? did it appear in other contexts? how critical is it? etc.

@johnwhitington
Copy link
Contributor

Still an issue.

Today I am debugging a program in Python, which uses a mixed C/OCaml DLL, and has stdout and stderr output written by all three languages. I can set stdout and stderr to be unbuffered in both C and Python, and then redirecting with >foo 2>&1 provides a chronological output mixing debug output with error output. But I cannot get OCaml stderr output in its correct place in the order, without adding explicit flushes to every stderr output in a very large OCaml program.

It's fixable, but it would be a lot easier to be able to control OCaml's stderr buffering.

@nojb
Copy link
Contributor

nojb commented Jul 23, 2021

Technically I don't see a problem adding set_channel_buffered : out_channel -> bool -> unit. Setting it to false would mean flushing the channel after every primitive output operation (caml_ml_output, caml_ml_output_char, etc). For backwards compatibility we will probably want to keep stderr buffered by default though.

@johnwhitington
Copy link
Contributor

Would it really affect backward compatibility to simply change stderr to always-unbuffered like other languages do? It is presently unspecified when flushing happens (except that it must happen at certain points), so the specification does not change. What would be the practical problems, if any?

@nojb
Copy link
Contributor

nojb commented Jul 23, 2021

Would it really affect backward compatibility to simply change stderr to always-unbuffered like other languages do? It is presently unspecified when flushing happens (except that it must happen at certain points), so the specification does not change. What would be the practical problems, if any?

I can think of two possible side-effects: changing the output interleaving between stdout and stderr and performance degradation when a lot of data is being output on stderr.

@johnwhitington
Copy link
Contributor

Thanks. Counterpoints: the interleaving is undefined, so should never be relied upon. Both C and C++ have stderr unbuffered by default, so I presume it's not a performance problem for them.

POSIX says:

"At program start-up, three streams are predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). When opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device."

(https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_05)

I'm perfectly happy with your proposed interface, I just thought we should explore fixing it more directly before committing.

@nojb
Copy link
Contributor

nojb commented Jul 23, 2021

Thanks. Counterpoints: the interleaving is undefined, so should never be relied upon. Both C and C++ have stderr unbuffered by default, so I presume it's not a performance problem for them.

POSIX says:

"At program start-up, three streams are predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). When opened, the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device."

(https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_05)

Do C and C++ set stdout to be unbuffered when the output is the interactive console?

@johnwhitington
Copy link
Contributor

johnwhitington commented Jul 23, 2021

For C++ according to https://en.cppreference.com/w/cpp/io/c/std_streams:

"Three text streams are predefined. These streams are implicitly opened and unoriented at program startup.

  1. Associated with the standard input stream, used for reading conventional input. At program startup, the stream is fully buffered if and only if the stream can be determined not to refer to an interactive device.
  2. Associated with the standard output stream, used for writing conventional output. At program startup, the stream is fully buffered if and only if the stream can be determined not to refer to an interactive device.
  3. Associated with the standard error stream, used for writing diagnostic output. At program startup, the stream is not fully buffered.

What consistutes an interactive device is implementation-defined."

That page also says:

"Although not mandated by POSIX, the UNIX convention is that stdin and stdout are line-buffered if associated with a terminal and stderr is unbuffered."

@nojb
Copy link
Contributor

nojb commented Jul 24, 2021

Thanks for the references. So in POSIX there are 3 possible stages of buffering: unbuffered, line-buffered (where the buffer is flushed on a newline) and fully buffered.

In OCaml all channels are currently fully buffered (with the caveat that certain output functions such as print_endline and print_newline flush the channel before returning).

@johnwhitington Would it be correct to say that you are suggesting is to make stderr unbuffered as soon as it corresponds to a terminal, and leave everything else as it is today?

@pmetzger
Copy link
Member

stderr is definitely conventionally unbuffered. The goal is that error output is rare, and should show up immediately, both so the programmer knows when a problem happened, and so that a crash will not deprive the programmer of debugging information.

@nojb
Copy link
Contributor

nojb commented Jul 24, 2021

Can continue the discussion in #10538

@nojb
Copy link
Contributor

nojb commented Sep 15, 2021

Fixed in #10538

@nojb nojb closed this as completed Sep 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants