Version française
Home     About     Download     Resources     Contact us    
Browse thread
OCaml runtime using too much memory in 64-bit Linux
[ Home ] [ Index: by date | by threads ]
[ Search: ]

[ Message by date: previous | next ] [ Message in thread: previous | next ] [ Thread: previous | next ]
Date: -- (:)
From: Lionel Elie Mamane <lionel@m...>
Subject: Re: [Caml-list] Re: OCaml runtime using too much memory in 64-bit Linux
On Wed, Nov 14, 2007 at 11:22:46AM -0500, Stefan Monnier wrote:
>> and uses a page table for this purpose, with a dense representation
>> (an array of bytes).  If the major heap areas are closely spaced, this
> [...]
>> For 32-bit platforms, this isn't much of a problem since the maximum
>> size of the page table is 1 megabytes.  For 64-bit platforms, the sky

> How about allocating this array of bytes via mmap and then leave it
> uninitialized (relying on POSIX's guarantee that it's already
> initialized to zeros)?
> This way you can easily have a 4GB "dense" table which doesn't use much
> RAM since most of the 4GB will be mapped (via copy-on-write) to the same
> "zero page".

I think this will fail on a GNU/Linux 2.6 system with
/proc/sys/vm/overcommit_memory set to 2, or any other system that
behaves as Linux with overcommit to 2. (Meaning, it actually reserves
place from the swap+ram pool, so that any mmapped/sbrk'd memory can
actually be used. It permits the kernel to guarantee that even if all
programs actually use all the memory they allocate, it will be able to
serve them - albeit slowly (swap use)).

In particular, the addressing space of a 64 bit machine is, well... 64
bits, by definition. For 4kiB = 2^12 B pages, one thus needs a table
of size 2^(64-12) = 2^52 bytes, that is 4 EB. That is, on any machine
with less than that of memory (and overcommit to 2), the program will
not run. Even at one bit (and not byte) per page, that is still
16PB...

Big pages don't get you out of the problem. 4MB pages only buy you a
factor 1024, that is 4PB and 16GB.


-- 
Lionel