Version française
Home     About     Download     Resources     Contact us    

This site is updated infrequently. For up-to-date information, please visit the new OCaml website at

Browse thread
[Caml-list] Wish List for Large Mutable Objects
[ Home ] [ Index: by date | by threads ]
[ Search: ]

[ Message by date: previous | next ] [ Message in thread: previous | next ] [ Thread: previous | next ]
Date: 2004-08-02 (02:37)
From: David McClain <dmcclain1@m...>
Subject: Re: [Caml-list] Wish List for Large Mutable Objects
Okay now... not trying to start any flame wars, but you "guys in the
trenches" so to speak seem a bit short on real life experience outside of
your own field.

I have a perfectly good running VM as user process library running right now
in C++ that allows for mixed array files, arbitrary offsets into the file
for various array pointers, and this is all transparent to the user just as
I indicated in my wish list for OCaml.

In more than 20 years of scientific data access and analysis I have only
seen uniform arrays, one per file, generated by neophytes. In just about
every case I can remember; NetCDF, HDF, FITS, RIF Wave Files, MPEG, etc.,
these are all compound object files. The trouble with the simple minded
approach of one array per file is that most data acquisitions will then end
up with dozens of component data files and it becomes a tracking nightmare
to keep them all coordinated. Not so if you permit compound document files.

With a language as rich and wonderful as OCaml, I really can't understand
your hostility to useful additions to the language. If you don't want to
play, you don't have to join my sandbox -- find another.

David McClain
Senior Corporate Scientist
Avisere, Inc.

+1.520.390.7738 (USA)

----- Original Message ----- 
From: "Brandon J. Van Every" <>
To: "caml" <>
Sent: Saturday, July 31, 2004 21:06
Subject: RE: [Caml-list] Wish List for Large Mutable Objects

> David McClain wrote:
> >
> > Something I would like to see appear in the OCaml libraries,
> > and I don't have it yet myself, is the use of Copy-on-Write
> > and Scatter-Gather applied to large mutable objects such as
> > BigArrays.
> >
> > [and some other things in other posts]
> OCaml is not a lazy language.  Is it reasonable to expect Bigarray to
> perform lazy copies, under any permutation of complexity you have in
> mind?
> It seems like what you really want is to design the memory management of
> an Operating System.  If your files are so huge that they don't fit in
> main memory, why aren't you willing to use virtual memory?
> Scatter-Gather DMA is a device driver level capability that's hardware
> dependent.  I don't see why a high level language like OCaml should be
> exposing that kind of functionality, and I'm not entirely sure if it
> should be doing it under the hood either.  If your OS doesn't have the
> kind of memory management you want, maybe you should modify an open
> source OS, like Linux or BSD Unix, to do what you want?
> You ask why Array1, Array2, Array3 should be special cases.  Well,
> clearly because they're the most common, and you can perform access
> optimizations for each of these common cases.  It seems that you are
> only thinking of ***BIG*** arrays, i.e. your problems and nobody else's.
> Lotsa people don't have your notion of 'big'.  Indeed, I don't
> personally care about arrays being particularly big.  100MB would be
> pretty darn big for what I do in game development right now.  I do care
> about their contents being unboxed.  If someone wanted to rename
> Bigarray to UnboxedArray, that would suit my own priorities just fine.
> I don't understand the "starting from zero" complaint, with respect to
> arbitrary file formats.  If the file format is arbitrarily structured,
> it is not an array.  You will have to read it some other way.  Arrays
> are, generally speaking, composed of uniform elements.  At least, that's
> how all of us pedal-to-the-metal guys view them.  I suppose high level
> language guys often define the word 'array' to mean anything they want,
> like a list or a hash table or a map or whatever, but I don't think they
> should.
> I don't see why your Scientific notion of an 'infinite array' should be
> a basic language interface.  What would be so difficult about building
> your favorite array windowing scheme on top of the basic fixed length
> components, and calling that a library?  Like 'InfiniteArray' or
> something.  Then you'd write some access functions in some syntax you
> like, it would behave the way you like, and for your problems you'd be
> good.  I've done similar things to perform addressing on icosahedrons,
> to try to regularize the mathematics of a tiling of it.  I don't bother
> the user about it, my functions just do some computing to make it all
> work under the hood.
> I do wish Bigarray handled heterogeneous C structures.  Homogeneous
> arrays impose some design and interop constraints.
> Finally, I'm told that the "%" in the names of called functions in the
> sources means that ocamlopt generates different, better code.  The C
> routines are ignored, they're only used for ocamlc.
> Cheers,               
> Brand*n Van Every               S*attle, WA
> Praise Be to the caml-list Bayesian filter! It blesseth
> my postings, it is evil crap!  evil crap!  Bigarray!
> Unboxed overhead group!  Wondering!  chant chant chant...
> -------------------
> To unsubscribe, mail Archives:
> Bug reports: FAQ:
> Beginner's list:

To unsubscribe, mail Archives:
Bug reports: FAQ:
Beginner's list: