Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wish: Increase IO_BUFFER_SIZE #5501

Closed
vicuna opened this issue Feb 5, 2012 · 2 comments
Closed

Wish: Increase IO_BUFFER_SIZE #5501

vicuna opened this issue Feb 5, 2012 · 2 comments
Assignees

Comments

@vicuna
Copy link

vicuna commented Feb 5, 2012

Original bug ID: 5501
Reporter: gerd
Assigned to: @damiendoligez
Status: closed (set by @damiendoligez on 2012-04-10T14:09:52Z)
Resolution: fixed
Priority: normal
Severity: trivial
Version: 3.12.1
Fixed in version: 4.00.0+dev
Category: runtime system and C interface
Monitored by: mehdi @ygrek

Bug description

I think IO_BUFFER_SIZE=4096 (for channel I/O) is nowadays a bit low. For current machines and OS, this should be increased, maybe to 64K. (It's a constant in byterun/io.c).

A few indications why this is reasonable:

  • Certain OS increased their pipe buffer size (e.g. Linux to 64K, but it
    grows beyond that if needed; FreeBSD and OSX to 16K, with the option to
    grow to 64K)
  • RAM bandwidth is continously increasing
  • Systems have more RAM than ever

Additional information

Recently I ran into a case where the producer of a pipe slowed the consumer down, because only 4K were written at a time. The pattern was especially bad because the producer wrote the results of computations and used the CPU to 100%, and the 4K blocks were emitted with a certain frequency. The consumer was poll-based, and the poll loop was running way to often, also at the same frequency. The result was that the consumer also used another core to almost 100%. The lesson to learn for me was that the consumer cannot do anything against this when the producer drives it the wrong way. Using more of the pipe buffer finally solved the problem (by running the pipe like "producer | dd bs=1M iflags=fullblock | consumer")

@vicuna
Copy link
Author

vicuna commented Apr 10, 2012

Comment author: @alainfrisch

I'm also in favor of increasing the default value for IO_BUFFER_SIZE (to something between 16kB and 64kB). Who's against?

@vicuna
Copy link
Author

vicuna commented Apr 10, 2012

Comment author: @damiendoligez

Fixed in version 4.00 (commit 12331) and trunk (commit 12332).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants