On Wed, Apr 29, 2009 at 04:29, Brighten Godfrey <pbg@cs.berkeley.edu> wrote:
> I know nothing about the internals of these libraries.  But, the program is
> continuously reading lines from the file.  Thus, isn't there about the same
> amount of memory on the heap just before the problem starts and just after
> the problem starts?  I guess it is plausible that somehow, closing the file
> and re-opening it triggers a bad interaction with the GC...
>
> But in comparison, using Str in the same way (i.e., compiling the regexp
> every time it is used) works fine.

Note that the effect of not precompiling the regular expressions is
not just the overhead of this computation, but also vastly greater
GC-pressure.

The current GC-settings in Pcre will trigger a full GC-cycle every 500
regular expressions allocated, i.e. would perform a full major
collection every 500 lines in your case.  This setting works fine for
just about any application I've seen, because virtually nobody has to
create patterns dynamically at rates so high that this matters.

Thus, try hoisting out the compilation of the regexp first...

Markus

-- 
Markus Mottl        http://www.ocaml.info        markus.mottl@gmail.com