On Wed, Feb 06, 2008 at 12:55:04PM +0100, Oliver Bandel wrote:
> Hello,
> 
> I should have changed the Subject to: "Shocking Performance!!!"
> 
> but then possibly the spam-filter would become active ;-)
> 
> 
> The performance dramatically increased now!
> 
> I first had about 3min34  on my dataset.
> After throwing out some of the "^"-using
> functions, the time was about 1min55.
> 
> Now, after I threw out the rest of that "^"-stuff
> (which btw. made more of the catanations then
> the first thrown out functions, but was not called
> as often as trhe other functions) I'm under 20 seconds!
> (17..18 seconds!)
> 
> That's amazing! :-)


well i'm pretty sure you could go down even further with your own
implementation of a buffer library.

the buffer library is actually pretty bad since it's actually just a
simple string. each time the buffer need to grow, the string is
reallocated and the previous one is copied to the new string.
and you got the 16mb limit (max_string_length) on 32bit.

if you implement a growing array of fixed sized string (4K for example),
you just don't need to copy data each time your buffer need to grow. I
suspect it might be even faster than the normal buffer in your case
(lots of data appending), but depends on what you do with your buffer
afterwards.

-- 
Vincent Hanquez