0001909: Unix.read isn't POSIX-confirmant even if the OS is

Notes
(0000189) administrator (administrator) 2003-10-31 10:18	>That doesn't work. A blocking section is not allowed to access the >Caml heap in any way (at least in multi-threaded programs). Right. Now that you mention this, I found a discussion of it at: http://pauillac.inria.fr/~aschmitt/cwn/2003.04.22.html#3 [^] Thus the remaining options are: 1. malloc an arbitrary-sized buffer (which might be bad if it's huge) 2. document that, unlike POSIX, Unix.read might not read all the requested bytes from a plain file even when the bytes are available. 3. if the number of bytes to read exceeds the fixed-size buffer, before doing the C read, use select to wait for some data to become available on the file descriptor. Once data is available, read should return promptly, so the read could be done after the leave_blocking_section. This will only cause an extra I/O operation when we're reading big chunks. If the chunk is big enough we might save a bunch of I/O operations by reading it all at once, so this might improve average performance. -- Tim Freeman tim@fungible.com GPG public key fingerprint ECDF 46F8 3B80 BB9E 575D 7180 76DF FE00 34B1 5C78 Computers don't like it when you anthropomorphize them. -- Chris Phoenix

(0000190) administrator (administrator) 2003-10-31 16:10	> The length limit is a consequence of reading the bytes into a buffer > before copying them, and I don't see any point in doing that. You > could do this just as well: > > Begin_root (buf); > numbytes = Long_val(len); > enter_blocking_section(); > ret = read(Int_val(fd), &Byte(buf, Long_val(ofs)), (int) numbytes); > leave_blocking_section(); > if (ret == -1) uerror("read", Nothing); > End_roots(); That doesn't work. A blocking section is not allowed to access the Caml heap in any way (at least in multi-threaded programs). -- Damien

(0006829) doligez (manager) 2012-01-27 15:47	select() doesn't guarantee that data is available on the file descriptor when read() is called, because the data might disappear (i.e. be removed by some other process) between the call to select() and the call to read(). In that case, we are left with a thread that blocks outside of a blocking section, a major problem for multithreaded programs. Our only realistic option is to document this behaviour.

(0006835) gerd (reporter) 2012-01-27 21:04	The POSIX text is not very exact. Traditonally, read() does not return fewer bytes than requested for regular files (independently of how you read the POSIX specs - all POSIX-based OS do this). There are other cases which are also not exactly specified (e.g. behaviour for devices or message-based channels), and here the Ocaml behaviour hurts more. E.g. you cannot receive a 64K Internet datagram - it is cut off at 16K. In Ocamlnet the chosen workaround is to use bigarrays as primary buffers, i.e. you have something like val mem_read : Unix.file_descr -> buffer -> int -> int -> int where type buffer = (char, int8_unsigned_el, c_layout) Array1.t The user has now the option to make the buffer as large as necessary. This solution has the nice property that you can even save one data copy if you can directly process the data in the bigarray buffer. If you copy the bigarray to a string (btw, this function is missing in the runtime), you have exactly the same overhead as the current solution for Unix.read. As bigarrays are malloc-ed memory, we do not run into the problem that the buffer can be moved around by the GC when triggered by another thread. So, my suggestion is: - make something like mem_read the fundamental operation, and expose it for users needing exact control - Unix.read would remain the same, and the issue is documented - include functions for copying char bigarray to string and vice versa This might imply some restructuring of the libraries, especially, the C part of Bigarrays would have to be moved to the normal stdlib. Btw, write, recv, send have similar problems.

Issue History
Date Modified	Username	Field	Change
2005-11-18 10:13	administrator	New Issue
2012-01-27 15:47	doligez	Note Added: 0006829
2012-01-27 21:04	gerd	Note Added: 0006835
2012-06-21 20:16	frisch	Category	OCaml general => OCaml otherlibs

Relationships