Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0004448OCamlOCaml generalpublic2007-11-14 13:432009-02-01 09:38
Reporterxleroy 
Assigned Toxleroy 
PrioritynoneSeverityminorReproducibilitysometimes
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0004448: excessively large page table on 64-bit platforms
DescriptionAs reported on the caml-list, some users of 64-bit platforms noticed large page tables being allocated, e.g. "Growing page table to 104640820 entries". The goal of this PR is to better understand the situations where this occurs. Please add notes to this PR describing in particular:

- Your platform: processor, OS version (esp. which Linux distro), kernel version.

- The output of the following command:
     grep USE_MMAP `ocamlc -where`/caml/config.h
TagsNo tags attached.
Attached Files

- Relationships

-  Notes
(0004307)
smimram (reporter)
2007-11-14 14:23

Here are two runs of liquidsoap:

$ OCAMLRUNPARAM="v=12" ./liquidsoap 'output.dummy(blank())'
Growing heap to 960k bytes
Growing page table to 11770701 entries
Growing heap to 1440k bytes

$ OCAMLRUNPARAM="v=12" ./liquidsoap 'output.dummy(blank())'
Growing heap to 960k bytes
Growing page table to 129384191 entries
Growing heap to 1440k bytes

The processor is an "AMD Athlon(tm) 64 Processor 3500+", running Debian testing, and the kernel is 2.6.22-2-amd64. ocaml comes from the 3.10.0-8 Debian package.

$ grep USE_MMAP `ocamlc -where`/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC
(0004308)
xleroy (administrator)
2007-11-14 14:58

While you're at it, and if your OS is Lunix, the output of the following command is interesting as well:

cat /proc/sys/kernel/randomize_va_space

(Thanks, Basile.)
(0004309)
smimram (reporter)
2007-11-14 15:09
edited on: 2007-11-14 15:12

Here it is:

$ cat /proc/sys/kernel/randomize_va_space
1

and by the way, the problem disappears if I

echo 0 > /proc/sys/kernel/randomize_va_space

(0004311)
adamc (reporter)
2007-11-14 15:30

Linux distro is Fedora 5.

$ uname -a
Linux [hostname] 2.6.18.8pavel_64bit 0000005 SMP PREEMPT Tue May 8 12:28:30 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

$ grep USE_MMAP `ocamlc -where`/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC

$ cat /proc/sys/kernel/randomize_va_space
1
(0004312)
Christophe Troestler (reporter)
2007-11-14 15:42

> I suspect some Linux distros that applied address randomization patches
> to the stock Linux kernel.

In fact, randomization seems to be in the stock kernel: I have compiled

  Linux 2.6.23.1 #1 SMP Tue Nov 13 22:17:50 CET 2007 x86_64 GNU/Linux

downloaded from kernel.org a few days ago (no patches applied) and I have

  $ cat /proc/sys/kernel/randomize_va_space
  1

Thanks for your work,
ChriS
(0004313)
mottl (reporter)
2007-11-14 16:48

According to Adam, who filed the initial bug report, turning off address randomization in the kernel makes the problem go away. Though this solution is good enough for us for the while being, it would certainly be helpful if we could find a more general solution. Some applications might be so sensitive that people don't want to turn off this important security feature.

I am not sure whether this might help, but it seems that Linux has at least some respect for the "start" argument to mmap, i.e. takes it as a hint where to place the mapping. By starting out with "NULL" for the initial mapping and then just storing the start of the page following the current mapping in a global variable for the next allocation looks like a very simple thing to try. It may not be a perfect solution, since the kernel is not required to return a mapping at this address, but it may be good enough and does not require a substantial rewrite of the page handling code.
(0004314)
Vladimir Shabanov (reporter)
2007-11-14 17:14

$ cat /proc/sys/kernel/randomize_va_space
1

$ grep USE_MMAP `ocamlc -where`/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC

System is Debian testing, 2.6.21-2-amd64 kernel.

native executable, works ok, output is always the same:
$ OCAMLRUNPARAM="v=12" ./_build/game.opt
Growing heap to 960k bytes
Growing page table to 72391 entries
Growing heap to 1440k bytes
Growing page table to 90522 entries

bytecode executable:
$ OCAMLRUNPARAM="v=12" ./_build/game
Initial stack limit: 8192k bytes
Growing gray_vals to 32k bytes
Growing heap to 960k bytes
Growing page table to 141518746 entries <-- this number vary from run to run
Growing heap to 1440k bytes

Problem disappears after
$ echo 0 > /proc/sys/kernel/randomize_va_space
page table entries count is small and no more vary from run to run.

native output with randomize_va_space=0:
Growing heap to 960k bytes
Growing page table to 89517 entries
Growing heap to 1440k bytes
Growing page table to 107540 entries

bytecode output with randomize_va_space=0:
Initial stack limit: 8192k bytes
Growing gray_vals to 32k bytes
Growing heap to 960k bytes
Growing page table to 71741 entries
Growing heap to 1440k bytes
Growing page table to 98490 entries
(0004315)
Vladimir Shabanov (reporter)
2007-11-14 17:22

Forgot the processor:
$ cat /proc/cpuinfo | grep "model name"
model name : AMD Athlon(tm) 64 Processor 3500+
(0004319)
mottl (reporter)
2007-11-14 23:13
edited on: 2007-11-14 23:16

I can confirm now that passing a hint to mmap seems to fix the problem in the OCaml-runtime, though I haven't been able to test yet whether this will generally work well. If you change the function caml_aligned_mmap in byterun/unix.c as follows:

char * caml_aligned_mmap (asize_t size, int modulo, void **block)
{
  char *raw_mem;
  uintnat aligned_mem;
  static char *last_addr = NULL;
  raw_mem = (char *) mmap(last_addr, size + Page_size, PROT_READ | PROT_WRITE,
                          MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
  if (raw_mem == MAP_FAILED) return NULL;
  last_addr = raw_mem + size + 2 * Page_size;
  *block = raw_mem;
  raw_mem += modulo; /* Address to be aligned */
  aligned_mem = (((uintnat) raw_mem / Page_size + 1) * Page_size);
  return (char *) (aligned_mem - modulo);
}

then the problem with exploding page tables seems to go away. In one case it seemed to me that one may have to add more than just two page tables to get nearby address space, but I unfortunately don't have time now to find out when this may be necessary. This is at least a starting point for a cheap solution that may work well enough until a better implementation of page table management is available.

(0004332)
xleroy (administrator)
2007-11-20 16:50

Markus's suggestion looks like a good temporary workaround. I applied it in the 3.10 release branch. Meanwhile, I'm experimenting with replacing the page table by a hash table, with good results so far. This is candidate for inclusion in the trunk after more testing.
(0004400)
xleroy (administrator)
2008-01-03 11:02

In CVS trunk, reimplemented the page table as a sparse data structure (hash table). Preliminary tests are OK, both correctness-wise and performance-wise. More testing is welcome.
(0004828)
xleroy (administrator)
2009-02-01 09:38

The new implementation of page tables in 3.11 seems to work. I'm closing this PR.

- Issue History
Date Modified Username Field Change
2007-11-14 13:43 xleroy New Issue
2007-11-14 13:43 xleroy Status new => feedback
2007-11-14 14:23 smimram Note Added: 0004307
2007-11-14 14:58 xleroy Note Added: 0004308
2007-11-14 15:09 smimram Note Added: 0004309
2007-11-14 15:12 toots Note Added: 0004310
2007-11-14 15:12 smimram Note Edited: 0004309
2007-11-14 15:14 toots Note Deleted: 0004310
2007-11-14 15:30 adamc Note Added: 0004311
2007-11-14 15:42 Christophe Troestler Note Added: 0004312
2007-11-14 16:48 mottl Note Added: 0004313
2007-11-14 17:14 Vladimir Shabanov Note Added: 0004314
2007-11-14 17:22 Vladimir Shabanov Note Added: 0004315
2007-11-14 23:13 mottl Note Added: 0004319
2007-11-14 23:16 mottl Note Edited: 0004319
2007-11-17 09:17 xleroy Assigned To => xleroy
2007-11-17 09:17 xleroy Priority normal => none
2007-11-17 09:17 xleroy Category OCamldoc => OCaml general
2007-11-20 16:50 xleroy Note Added: 0004332
2007-11-20 16:50 xleroy Status feedback => assigned
2008-01-03 11:02 xleroy Note Added: 0004400
2008-01-03 11:02 xleroy Status assigned => resolved
2008-01-03 11:02 xleroy Resolution open => fixed
2008-01-03 11:02 xleroy Product Version 3.10.0 =>
2009-02-01 09:38 xleroy Note Added: 0004828
2009-02-01 09:38 xleroy Status resolved => closed


Copyright © 2000 - 2011 MantisBT Group
Powered by Mantis Bugtracker