On Apr 28, 2009, at 11:37 PM, Alain Frisch wrote:
> Brighten Godfrey wrote:
>> That occurred to me too, but there is no swapping.  The process  
>> uses less than 40 MB of memory.  Also, this wouldn't explain why it  
>> suddenly becomes slow exactly when it starts parsing the file the  
>> second time.
>
> This point is interesting: it is where PCRE starts compiling a lot  
> of regexps while the OCaml heap is already full of non-garbage value  
> (the value bound to first). For each allocated regexp, the OCaml  
> PCRE binding pushes some pressure on the OCaml GC; maybe, then, the  
> GC runs too often for the amount of memory it can really release.


I know nothing about the internals of these libraries.  But, the  
program is continuously reading lines from the file.  Thus, isn't  
there about the same amount of memory on the heap just before the  
problem starts and just after the problem starts?  I guess it is  
plausible that somehow, closing the file and re-opening it triggers a  
bad interaction with the GC...

But in comparison, using Str in the same way (i.e., compiling the  
regexp every time it is used) works fine.

Thanks,
~Brighten