You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Original bug ID: 5501 Reporter: gerd Assigned to:@damiendoligez Status: closed (set by @damiendoligez on 2012-04-10T14:09:52Z) Resolution: fixed Priority: normal Severity: trivial Version: 3.12.1 Fixed in version: 4.00.0+dev Category: runtime system and C interface Monitored by: mehdi @ygrek
Bug description
I think IO_BUFFER_SIZE=4096 (for channel I/O) is nowadays a bit low. For current machines and OS, this should be increased, maybe to 64K. (It's a constant in byterun/io.c).
A few indications why this is reasonable:
Certain OS increased their pipe buffer size (e.g. Linux to 64K, but it
grows beyond that if needed; FreeBSD and OSX to 16K, with the option to
grow to 64K)
RAM bandwidth is continously increasing
Systems have more RAM than ever
Additional information
Recently I ran into a case where the producer of a pipe slowed the consumer down, because only 4K were written at a time. The pattern was especially bad because the producer wrote the results of computations and used the CPU to 100%, and the 4K blocks were emitted with a certain frequency. The consumer was poll-based, and the poll loop was running way to often, also at the same frequency. The result was that the consumer also used another core to almost 100%. The lesson to learn for me was that the consumer cannot do anything against this when the producer drives it the wrong way. Using more of the pipe buffer finally solved the problem (by running the pipe like "producer | dd bs=1M iflags=fullblock | consumer")
The text was updated successfully, but these errors were encountered:
Original bug ID: 5501
Reporter: gerd
Assigned to: @damiendoligez
Status: closed (set by @damiendoligez on 2012-04-10T14:09:52Z)
Resolution: fixed
Priority: normal
Severity: trivial
Version: 3.12.1
Fixed in version: 4.00.0+dev
Category: runtime system and C interface
Monitored by: mehdi @ygrek
Bug description
I think IO_BUFFER_SIZE=4096 (for channel I/O) is nowadays a bit low. For current machines and OS, this should be increased, maybe to 64K. (It's a constant in byterun/io.c).
A few indications why this is reasonable:
grows beyond that if needed; FreeBSD and OSX to 16K, with the option to
grow to 64K)
Additional information
Recently I ran into a case where the producer of a pipe slowed the consumer down, because only 4K were written at a time. The pattern was especially bad because the producer wrote the results of computations and used the CPU to 100%, and the 4K blocks were emitted with a certain frequency. The consumer was poll-based, and the poll loop was running way to often, also at the same frequency. The result was that the consumer also used another core to almost 100%. The lesson to learn for me was that the consumer cannot do anything against this when the producer drives it the wrong way. Using more of the pipe buffer finally solved the problem (by running the pipe like "producer | dd bs=1M iflags=fullblock | consumer")
The text was updated successfully, but these errors were encountered: