Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005201OCamlOCamlbuild (the tool)public2011-01-04 16:292014-07-16 20:54
Reporterdaweil 
Assigned To 
PrioritynormalSeveritytweakReproducibilityalways
StatusconfirmedResolutionopen 
PlatformWindowsOSXPOS Version2
Product Version3.12.0 
Target Versionafter-4.02.0Fixed in Version 
Summary0005201: Performance issue for ocamlbuild on Windows
Descriptionon Windows system, each command executed by ocambuild is encapsulated in a "bash -C" command which has a non negligeable overhead (~ 0.5s).
Unless the Filename.quote function is fixed, a simple improvement is to change a line in file ocamlbuild/my_std.ml,
In function sys_command, just replace the line
 let cmd = "bash -c "^Filename.quote cmd in

by
  let cmd = "sh -c "^Filename.quote cmd in

ocamlbuild will go much faster on Windows.

TagsNo tags attached.
Attached Files

- Relationships
related to 0004981resolvedxclerc ocamlbuild/my_std.ml shouldn't call bash 

-  Notes
(0005759)
ertai (developer)
2011-01-05 16:19

In the same vein I would also be in favor of not calling the shell when the arguments are either easily quotable or need not to be quoted.
(0008124)
daweil (reporter)
2012-09-20 14:11

and because ocamlbuild call the "bash" or the "sh" shell, ocamlbuild does'nt work when call direcly called from a DOS shell.
So this issue is not only a performance issue.
(0008125)
protz (manager)
2012-09-20 14:17

Yes, however, the recommend way to run OCaml under Windows is inside a Cygwin environment (either with the cygwin-packaged ocaml, or the native win32 ocaml compilers) so that you have a shell and the gnu tools available to work with. Argument quoting is such a mess on windows that I don't think it would be easy to make sure ocamlbuild can rely on cmd.exe under windows...
(0008126)
ygrek (reporter)
2012-09-20 15:04

FTR see the discussion and referenced articles at https://forge.ocamlcore.org/tracker/?func=detail&aid=1170&group_id=54&atid=291 [^]
(0008615)
Camarade_Tux (reporter)
2012-12-15 18:05
edited on: 2012-12-15 18:12

I've started to look at the issue a bit more in-depth.

ocamlbuild uses this for different kinds of commands:
1- calls to cp, rm, mkdir, find, ...
2- any command through "run_and_read" or "run_and_open" functions which start a process and retrieve its standard output

1- is probably the biggest performance killer by far, especially on windows. Spawning bash on msys or cygwin takes time and spawning another command will only be slower. These should be pretty easy to implement.

2- is more annoying. The functions "run_and_*" seem to be usable by plugins but maybe not officially (can someone confirm this?). Overall it would be nicer to avoid such functions or reduce their number (some are easy: there's a call to readlink).

There will still be a few calls ("ocamlfind list" for instance) anyway. In ygrek's link, thelema has implemented the necessary quoting in ocaml; would it be possible to include this in the standard library? Unix.exec* functions should also be rewritten to use that quoting.

Moving the run_and_* functions from ocamlbuild to the standard library would probably be good too since they're already in the ocaml sources and are needed by projects which cannot easily use an additional library (ocamlbuild, my yypkg package manager, probably omake, ocamlfind, godi, ...).

So, does removing as many calls to external commands as possible and adding support for proper quoting for processes in the standard library sound like a good way?

PS: this issue should be made linked to "0004981: ocamlbuild/my_std.ml shouldn't call bash" in mantis.

(0009978)
daweil (reporter)
2013-07-29 14:05

I made some more test this last version of ocaml (4.01).
- 30 s to compile my code on Windows, 10s on linux on the same machine.
- With using "sh" instead of "bash" (or "bash --norc"), 20s to compile on Windows which is better thow twice slower than on Linux.

So, if bash cannot be removed, why not patching my_std.ml by "let cmd = "bash --norc -c "^Filename.quote cmd in" ?
(0009991)
gasche (developer)
2013-07-29 18:07

I know nothing about Windows, so please bear with me if I have stupid
questions. I would be ready to add --norc as proposed (while removing
bash seems fishy), but I don't understand why this is need. My "man
bash" says:

> An interactive shell is one started without non-option arguments and
> without the -c option whose standard input and error are both
> connected to terminals (as determined by isatty(3)), or one started
> with the -i option.

[...]

> When an interactive shell that is not a login shell is started, bash
> reads and executes commands from /etc/bash.bashrc and ~/.bashrc, if
> these files exist. This may be inhibited by using the --norc
> option.

My understanding is that by passing option -c as ocamlbuild already
does, we are not in the "interactive shell" case and --norc should not
be necessary. Am I wrong in this reading of the manual? Does
adding --norc really make a performance difference anyway?
(0010016)
Camarade_Tux (reporter)
2013-07-30 20:26

Btw, quoting bash's manpage:
  ~/.bashrc if the shell is interactive. This option is on by default if the shell is invoked as sh.


Back to removing the need for bash, I've read specs and explanations more and I believe the only way is to rely on Unix.create_process* and not others.
Most of the process-spawning functions in ocaml call the system shell and going through cmd.exe seems quite pointless to me (or is the shell actually relied upon?) and currently, ocaml's lib has no function to do the kind of escaping that cmd.exe requires, leading to the use of bash to parse the command-line and escape arguments properly before calling our actual command.
(0010059)
gasche (developer)
2013-08-01 11:55

I'm waiting for feedback from daweil on whether --norc really makes a difference.
(0010061)
daweil (reporter)
2013-08-01 12:07

Yes, the --norc options makes a difference on Windows, but only 30% better, cf my note on July 29th.
But of course, if you can remove the call to bash, performance on Windows && Linux should be identitical. And the blog mentionned in http://caml.inria.fr/mantis/view.php?id=4981 [^] : http://blogs.msdn.com/b/twistylittlepassagesallalike/archive/2011/04/23/everyone-quotes-arguments-the-wrong-way.aspx [^]
could help to quote commands correctly for Windows instead of calling bash.
(0010095)
gasche (developer)
2013-08-04 22:00

I committed the addition of the --norc option. I'm leaving the bug open as a place to welcome patches to remove some of the bash invocations in ocamlbuild (hint hint).

- Issue History
Date Modified Username Field Change
2011-01-04 16:29 daweil New Issue
2011-01-05 16:19 ertai Note Added: 0005759
2011-01-05 16:19 ertai Assigned To => xclerc
2011-01-05 16:19 ertai Status new => assigned
2012-02-02 15:17 protz Category OCamlbuild => OCamlbuild (the tool)
2012-07-10 17:17 doligez Target Version => 4.01.0+dev
2012-07-31 13:36 doligez Target Version 4.01.0+dev => 4.00.1+dev
2012-09-19 14:57 doligez Severity minor => tweak
2012-09-19 14:57 doligez Target Version 4.00.1+dev => 4.01.0+dev
2012-09-20 14:11 daweil Note Added: 0008124
2012-09-20 14:17 protz Note Added: 0008125
2012-09-20 15:04 ygrek Note Added: 0008126
2012-12-15 18:05 Camarade_Tux Note Added: 0008615
2012-12-15 18:12 Camarade_Tux Note Edited: 0008615 View Revisions
2013-07-28 23:19 gasche Relationship added related to 0004981
2013-07-29 14:05 daweil Note Added: 0009978
2013-07-29 18:07 gasche Note Added: 0009991
2013-07-30 20:26 Camarade_Tux Note Added: 0010016
2013-08-01 11:55 gasche Note Added: 0010059
2013-08-01 11:55 gasche Assigned To xclerc =>
2013-08-01 11:55 gasche Status assigned => feedback
2013-08-01 12:07 daweil Note Added: 0010061
2013-08-01 12:07 daweil Status feedback => new
2013-08-04 22:00 gasche Note Added: 0010095
2013-08-04 22:00 gasche Status new => confirmed
2013-08-14 09:23 doligez Target Version 4.01.0+dev => 4.01.1+dev
2014-05-25 20:20 doligez Target Version 4.01.1+dev => 4.02.0+dev
2014-07-16 20:54 doligez Target Version 4.02.0+dev => after-4.02.0


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker