Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

excessively large page table on 64-bit platforms #4448

Closed
vicuna opened this issue Nov 14, 2007 · 12 comments
Closed

excessively large page table on 64-bit platforms #4448

vicuna opened this issue Nov 14, 2007 · 12 comments
Assignees
Labels

Comments

@vicuna
Copy link

vicuna commented Nov 14, 2007

Original bug ID: 4448
Reporter: @xavierleroy
Assigned to: @xavierleroy
Status: closed (set by @xavierleroy on 2009-02-01T08:38:06Z)
Resolution: fixed
Severity: minor
Category: ~DO NOT USE (was: OCaml general)
Monitored by: "Vladimir Shabanov" adamc toots till @Chris00 smimram @oandrieu @yakobowski @alainfrisch @mmottl

Bug description

As reported on the caml-list, some users of 64-bit platforms noticed large page tables being allocated, e.g. "Growing page table to 104640820 entries". The goal of this PR is to better understand the situations where this occurs. Please add notes to this PR describing in particular:

  • Your platform: processor, OS version (esp. which Linux distro), kernel version.

  • The output of the following command:
    grep USE_MMAP ocamlc -where/caml/config.h

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: smimram

Here are two runs of liquidsoap:

$ OCAMLRUNPARAM="v=12" ./liquidsoap 'output.dummy(blank())'
Growing heap to 960k bytes
Growing page table to 11770701 entries
Growing heap to 1440k bytes

$ OCAMLRUNPARAM="v=12" ./liquidsoap 'output.dummy(blank())'
Growing heap to 960k bytes
Growing page table to 129384191 entries
Growing heap to 1440k bytes

The processor is an "AMD Athlon(tm) 64 Processor 3500+", running Debian testing, and the kernel is 2.6.22-2-amd64. ocaml comes from the 3.10.0-8 Debian package.

$ grep USE_MMAP ocamlc -where/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: @xavierleroy

While you're at it, and if your OS is Lunix, the output of the following command is interesting as well:

cat /proc/sys/kernel/randomize_va_space

(Thanks, Basile.)

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: smimram

Here it is:

$ cat /proc/sys/kernel/randomize_va_space
1

and by the way, the problem disappears if I

echo 0 > /proc/sys/kernel/randomize_va_space

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: adamc

Linux distro is Fedora 5.

$ uname -a
Linux [hostname] 2.6.18.8pavel_64bit #5 SMP PREEMPT Tue May 8 12:28:30 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

$ grep USE_MMAP ocamlc -where/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC

$ cat /proc/sys/kernel/randomize_va_space
1

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: @Chris00

I suspect some Linux distros that applied address randomization patches
to the stock Linux kernel.

In fact, randomization seems to be in the stock kernel: I have compiled

Linux 2.6.23.1 #1 SMP Tue Nov 13 22:17:50 CET 2007 x86_64 GNU/Linux

downloaded from kernel.org a few days ago (no patches applied) and I have

$ cat /proc/sys/kernel/randomize_va_space
1

Thanks for your work,
ChriS

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: @mmottl

According to Adam, who filed the initial bug report, turning off address randomization in the kernel makes the problem go away. Though this solution is good enough for us for the while being, it would certainly be helpful if we could find a more general solution. Some applications might be so sensitive that people don't want to turn off this important security feature.

I am not sure whether this might help, but it seems that Linux has at least some respect for the "start" argument to mmap, i.e. takes it as a hint where to place the mapping. By starting out with "NULL" for the initial mapping and then just storing the start of the page following the current mapping in a global variable for the next allocation looks like a very simple thing to try. It may not be a perfect solution, since the kernel is not required to return a mapping at this address, but it may be good enough and does not require a substantial rewrite of the page handling code.

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: Vladimir Shabanov

$ cat /proc/sys/kernel/randomize_va_space
1

$ grep USE_MMAP ocamlc -where/caml/config.h
#define USE_MMAP_INSTEAD_OF_MALLOC

System is Debian testing, 2.6.21-2-amd64 kernel.

native executable, works ok, output is always the same:
$ OCAMLRUNPARAM="v=12" ./_build/game.opt
Growing heap to 960k bytes
Growing page table to 72391 entries
Growing heap to 1440k bytes
Growing page table to 90522 entries

bytecode executable:
$ OCAMLRUNPARAM="v=12" ./_build/game
Initial stack limit: 8192k bytes
Growing gray_vals to 32k bytes
Growing heap to 960k bytes
Growing page table to 141518746 entries <-- this number vary from run to run
Growing heap to 1440k bytes

Problem disappears after
$ echo 0 > /proc/sys/kernel/randomize_va_space
page table entries count is small and no more vary from run to run.

native output with randomize_va_space=0:
Growing heap to 960k bytes
Growing page table to 89517 entries
Growing heap to 1440k bytes
Growing page table to 107540 entries

bytecode output with randomize_va_space=0:
Initial stack limit: 8192k bytes
Growing gray_vals to 32k bytes
Growing heap to 960k bytes
Growing page table to 71741 entries
Growing heap to 1440k bytes
Growing page table to 98490 entries

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: Vladimir Shabanov

Forgot the processor:
$ cat /proc/cpuinfo | grep "model name"
model name : AMD Athlon(tm) 64 Processor 3500+

@vicuna
Copy link
Author

vicuna commented Nov 14, 2007

Comment author: @mmottl

I can confirm now that passing a hint to mmap seems to fix the problem in the OCaml-runtime, though I haven't been able to test yet whether this will generally work well. If you change the function caml_aligned_mmap in byterun/unix.c as follows:

char * caml_aligned_mmap (asize_t size, int modulo, void **block)
{
char *raw_mem;
uintnat aligned_mem;
static char *last_addr = NULL;
raw_mem = (char *) mmap(last_addr, size + Page_size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (raw_mem == MAP_FAILED) return NULL;
last_addr = raw_mem + size + 2 * Page_size;
block = raw_mem;
raw_mem += modulo; /
Address to be aligned */
aligned_mem = (((uintnat) raw_mem / Page_size + 1) * Page_size);
return (char *) (aligned_mem - modulo);
}

then the problem with exploding page tables seems to go away. In one case it seemed to me that one may have to add more than just two page tables to get nearby address space, but I unfortunately don't have time now to find out when this may be necessary. This is at least a starting point for a cheap solution that may work well enough until a better implementation of page table management is available.

@vicuna
Copy link
Author

vicuna commented Nov 20, 2007

Comment author: @xavierleroy

Markus's suggestion looks like a good temporary workaround. I applied it in the 3.10 release branch. Meanwhile, I'm experimenting with replacing the page table by a hash table, with good results so far. This is candidate for inclusion in the trunk after more testing.

@vicuna
Copy link
Author

vicuna commented Jan 3, 2008

Comment author: @xavierleroy

In CVS trunk, reimplemented the page table as a sparse data structure (hash table). Preliminary tests are OK, both correctness-wise and performance-wise. More testing is welcome.

@vicuna
Copy link
Author

vicuna commented Feb 1, 2009

Comment author: @xavierleroy

The new implementation of page tables in 3.11 seems to work. I'm closing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants