For the OCaml packages on ppc64/ppc64le, we keep having bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1204876
The OCaml compiler is quite recursive, and so it can easily overflow the default stack. For reasons that are not entirely clear this happens only on ppc64/ppc64le (not on x86 or aarch64). Perhaps POWER stack frames are bigger, or the default stack size is smaller.
One way to "fix" this would be to modify every single ocaml-* spec file to add:
%ifarch %{power64} ulimit -s 65536 %endif
However that doesn't fix the problem for people running ocamlopt on their own programs. The issue was discussed upstream, but upstream don't want to make the compiler stages less mutually recursive since (a) it makes the code far more complex and (b) stack memory is just a special use-case for ordinary memory so why should they?
Could we increase the default stack size on ppc64?
Or are there other alternatives? (I thought briefly about carrying a downstream compiler patch which does the equivalent of 'ulimit -s 65536' inside the compiler).
Rich.
On Tue, 2015-03-24 at 12:19 +0000, Richard W.M. Jones wrote:
For the OCaml packages on ppc64/ppc64le, we keep having bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1204876
The OCaml compiler is quite recursive, and so it can easily overflow the default stack. For reasons that are not entirely clear this happens only on ppc64/ppc64le (not on x86 or aarch64). Perhaps POWER stack frames are bigger, or the default stack size is smaller.
The default thread stack is 8MB-64KB plus a 64KB guard page. PPC uses a 64K default page size.
So there are two possible problems:
1) running into the guard page for a single thread. 2) the sum of all thread stacks exceeds the ulimit -s
If changing the ulimit -s solves the problem the sum of all thread stacks has exceeded the ulimit.
If in cases where increasing the ulimit -s to unlimited does no help then you have at least one thread that has overflowed the default stack.
Power has lots of registers (32x64b GPRs + 32x64b FPRs + 32x128b VRs) so we are likely to need a bigger stack frame at each level. The minimums are 112 bytes for ELF V1 ABI (PPC64BE) and 48 for ELF V2 ABI (PPC64LE), but any interesting function will need additional space to spill non-volatiles across function calls and store any local variables.
You (ocaml runtime) can control the stack size on a per thread basis via pthread_attr_setstack().
I am not sure how ocaml is generating code for PPC64, you could look in to split stack support, but at this time GCC does not implement split stack.
One way to "fix" this would be to modify every single ocaml-* spec file to add:
%ifarch %{power64} ulimit -s 65536 %endif
However that doesn't fix the problem for people running ocamlopt on their own programs. The issue was discussed upstream, but upstream don't want to make the compiler stages less mutually recursive since (a) it makes the code far more complex and (b) stack memory is just a special use-case for ordinary memory so why should they?
Could we increase the default stack size on ppc64?
Or are there other alternatives? (I thought briefly about carrying a downstream compiler patch which does the equivalent of 'ulimit -s 65536' inside the compiler).
Rich.
On Tue, Mar 24, 2015 at 11:35:30AM -0500, Steven Munroe wrote:
On Tue, 2015-03-24 at 12:19 +0000, Richard W.M. Jones wrote:
For the OCaml packages on ppc64/ppc64le, we keep having bugs like this one: https://bugzilla.redhat.com/show_bug.cgi?id=1204876
The OCaml compiler is quite recursive, and so it can easily overflow the default stack. For reasons that are not entirely clear this happens only on ppc64/ppc64le (not on x86 or aarch64). Perhaps POWER stack frames are bigger, or the default stack size is smaller.
The default thread stack is 8MB-64KB plus a 64KB guard page. PPC uses a 64K default page size.
So there are two possible problems:
- running into the guard page for a single thread.
- the sum of all thread stacks exceeds the ulimit -s
The OCaml compiler is single threaded AFAIK.
If changing the ulimit -s solves the problem the sum of all thread stacks has exceeded the ulimit.
If in cases where increasing the ulimit -s to unlimited does no help then you have at least one thread that has overflowed the default stack.
Power has lots of registers (32x64b GPRs + 32x64b FPRs + 32x128b VRs) so we are likely to need a bigger stack frame at each level. The minimums are 112 bytes for ELF V1 ABI (PPC64BE) and 48 for ELF V2 ABI (PPC64LE), but any interesting function will need additional space to spill non-volatiles across function calls and store any local variables.
You (ocaml runtime) can control the stack size on a per thread basis via pthread_attr_setstack().
I am not sure how ocaml is generating code for PPC64, you could look in to split stack support,
OCaml uses its own code generator. However the description of -fsplit-stack from GCC sounds interesting. Are there any more details of how exactly it works? Does it catch the segfault from hitting the guard page and do something clever?
but at this time GCC does not implement split stack.
Did you mean "does implement"? GCC 5 documents it at least ...
Rich.
On Wed, Mar 25, 2015 at 05:54:58PM +0000, Richard W.M. Jones wrote:
OCaml uses its own code generator. However the description of -fsplit-stack from GCC sounds interesting. Are there any more details of how exactly it works? Does it catch the segfault from hitting the guard page and do something clever?
Just after posting that I found the details:
https://gcc.gnu.org/wiki/SplitStacks
Rich.
On Tue, 2015-03-24 at 11:35 -0500, Steven Munroe wrote:
I am not sure how ocaml is generating code for PPC64, you could look in to split stack support, but at this time GCC does not implement split stack.
... for PPC64.
I wouldn't want to do it in OCaml before it's supported in GCC and the runtime. But once it *is*, it shouldn't be hard to make OCaml support it.
It's mostly just a matter of emitting the right instructions in the function prologue and epilogue, in emit.mlp.
But it does depend on the runtime support for allocating more stack while not overflowing the 'slop' space on the existing stack, and linker support for expanding the stack frame size when calling through to legacy non-split-stack functions, and probably other things. So not something we'd want to do purely within OCaml.
I'm a little confused about what the problem is, though.
If the compiler is single-threaded, and increasing the stack ulimit fixes the problem, that implies that the default stack ulimit is less than the 8MiB-64KiB that it takes to reach the guard page... can that be right? Richard, what does 'ulimit -s' report *before* you increase it?
On Wed, Mar 25, 2015 at 06:30:25PM +0000, David Woodhouse wrote:
On Tue, 2015-03-24 at 11:35 -0500, Steven Munroe wrote:
I am not sure how ocaml is generating code for PPC64, you could look in to split stack support, but at this time GCC does not implement split stack.
... for PPC64.I wouldn't want to do it in OCaml before it's supported in GCC and the runtime. But once it *is*, it shouldn't be hard to make OCaml support it.
It's mostly just a matter of emitting the right instructions in the function prologue and epilogue, in emit.mlp.
But it does depend on the runtime support for allocating more stack while not overflowing the 'slop' space on the existing stack, and linker support for expanding the stack frame size when calling through to legacy non-split-stack functions, and probably other things. So not something we'd want to do purely within OCaml.
I'm a little confused about what the problem is, though.
In summary: when you run ocamlopt to compile a sufficiently complicated OCaml program, ocamlopt segfaults. We found the cure is to do 'ulimit -s 65536' before running ocamlopt.
If the compiler is single-threaded, and increasing the stack ulimit fixes the problem, that implies that the default stack ulimit is less than the 8MiB-64KiB that it takes to reach the guard page... can that be right? Richard, what does 'ulimit -s' report *before* you increase it?
$ ulimit -s 8192
(For reference, all the limits are attached below).
That is from Fedora 20 ppc64 (not -le). The server is a POWER 7 LPAR. The page size is 64K.
I don't currently have access to a newer Fedora machine, but the same segfault was reported in current Rawhide and it was cured in the same way, so it seems unlikely that the default stack size is different there.
Rich.
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 496059 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 4096 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
On Wed, Mar 25, 2015 at 06:30:25PM +0000, David Woodhouse wrote:
If the compiler is single-threaded, and increasing the stack ulimit fixes the problem, that implies that the default stack ulimit is less than the 8MiB-64KiB that it takes to reach the guard page...
Just so I'm clear, is the stack supposed to grow down automatically (ie. does the stack automatically use MAP_GROWSDOWN), or is OCaml supposed to do something when the stack hits the guard page?
I guess it's also possible that an OCaml stack frame is so big that it skips the guard page, or that GCC does some kind of stack "filling" to trigger the guard page which OCaml does not do.
Rich.
On Wed, 2015-03-25 at 19:45 +0000, Richard W.M. Jones wrote:
On Wed, Mar 25, 2015 at 06:30:25PM +0000, David Woodhouse wrote:
If the compiler is single-threaded, and increasing the stack ulimit fixes the problem, that implies that the default stack ulimit is less than the 8MiB-64KiB that it takes to reach the guard page...
Just so I'm clear, is the stack supposed to grow down automatically (ie. does the stack automatically use MAP_GROWSDOWN), or is OCaml supposed to do something when the stack hits the guard page?
I guess it's also possible that an OCaml stack frame is so big that it skips the guard page, or that GCC does some kind of stack "filling" to trigger the guard page which OCaml does not do.
This is easy to trigger, right? Can you catch it in gdb and 't a a bt'
On Wed, Mar 25, 2015 at 08:03:31PM +0000, David Woodhouse wrote:
On Wed, 2015-03-25 at 19:45 +0000, Richard W.M. Jones wrote:
On Wed, Mar 25, 2015 at 06:30:25PM +0000, David Woodhouse wrote:
If the compiler is single-threaded, and increasing the stack ulimit fixes the problem, that implies that the default stack ulimit is less than the 8MiB-64KiB that it takes to reach the guard page...
Just so I'm clear, is the stack supposed to grow down automatically (ie. does the stack automatically use MAP_GROWSDOWN), or is OCaml supposed to do something when the stack hits the guard page?
I guess it's also possible that an OCaml stack frame is so big that it skips the guard page, or that GCC does some kind of stack "filling" to trigger the guard page which OCaml does not do.
This is easy to trigger, right? Can you catch it in gdb and 't a a bt'
I won't bore you with the full stack trace, but the top and bottom and some interesting registers are below.
It confirms a few things:
- Stack is overflowing because of a highly recursive function (which does eventually bottom-out, if you give it a big enough stack).
- Compiler is single threaded.
- %r1 (at top) - %r1 (at bottom) is approximately 8 MB
- Avg. size of each stack frame is ~ 168 bytes.
Rich.
On Wed, 2015-03-25 at 19:45 +0000, Richard W.M. Jones wrote:
On Wed, Mar 25, 2015 at 06:30:25PM +0000, David Woodhouse wrote:
If the compiler is single-threaded, and increasing the stack ulimit fixes the problem, that implies that the default stack ulimit is less than the 8MiB-64KiB that it takes to reach the guard page...
Just so I'm clear, is the stack supposed to grow down automatically (ie. does the stack automatically use MAP_GROWSDOWN), or is OCaml supposed to do something when the stack hits the guard page?
It depends on which code is allocating the stack.
The default GLIBC implementation under pthread_create will allocate it as single anonymous mmap of 8MB then use mprotect on the lowest page to mark the guard page no-access. As in:
3fff8ab40000-3fff8ab50000 ---p 00000000 00:00 0 3fff8ab50000-3fff8b340000 rw-p 00000000 00:00 [stack: 30348] 3fff8b340000-3fff8b350000 ---p 00000000 00:00 0 3fff8b350000-3fff8bb40000 rw-p 00000000 00:00 0 3fff8bb40000-3fff8bb50000 ---p 00000000 00:00 0 3fff8bb50000-3fff8c340000 rw-p 00000000 00:00 0 3fff8c340000-3fff8c350000 ---p 00000000 00:00 0 3fff8c350000-3fff8cb40000 rw-p 00000000 00:00 0
I guess it's also possible that an OCaml stack frame is so big that it skips the guard page, or that GCC does some kind of stack "filling" to trigger the guard page which OCaml does not do.
Rich.
On Wed, Mar 25, 2015 at 07:57:25PM -0500, Steven Munroe wrote:
On Wed, 2015-03-25 at 19:45 +0000, Richard W.M. Jones wrote:
On Wed, Mar 25, 2015 at 06:30:25PM +0000, David Woodhouse wrote:
If the compiler is single-threaded, and increasing the stack ulimit fixes the problem, that implies that the default stack ulimit is less than the 8MiB-64KiB that it takes to reach the guard page...
Just so I'm clear, is the stack supposed to grow down automatically (ie. does the stack automatically use MAP_GROWSDOWN), or is OCaml supposed to do something when the stack hits the guard page?
It depends on which code is allocating the stack.
The default GLIBC implementation under pthread_create will allocate it as single anonymous mmap of 8MB then use mprotect on the lowest page to mark the guard page no-access. As in:
Is pthread_create used even for the main thread in the process (I thought the kernel created that). In any case threads aren't being used explicitly by this process.
Anyway I guess the guard page is just a hole that causes the process to abort (as observed) and is not a mechanism for automatically growing the stack.
Rich.
On Thu, 2015-03-26 at 08:57 +0000, Richard W.M. Jones wrote:
Is pthread_create used even for the main thread in the process (I thought the kernel created that). In any case threads aren't being used explicitly by this process.
Anyway I guess the guard page is just a hole that causes the process to abort (as observed) and is not a mechanism for automatically growing the stack.
Right, It's a mechanism for catching stack overflow reliably, rather than letting it cause random memory corruption.
But for the original stack created by the kernel when the process is created, I think you just don't have the guard page. All that stops you is the ulimit.
So... we could potentially look for ways to reduce the stack usage of OCaml-compiled code, or you get to keep increasing the limit.
It might be worth taking a look at the stack frames and seeing if I did anything entirely stupid when setting them up, and whether they can be made smaller. Although if you said it was averaging ~168 bytes, that doesn't seem *particularly* excessive. Can you see how big the stack actually does get in your worst case, and how much we'd have to shrink by in order to get away without tweaking the ulimit?
Perhaps we could also look at using a smaller stack frame between OCaml functions. If we're never using certain registers in the OCaml code generator, we don't need to save them or leave space for them on the stack. All we need to do it ensure that if we're calling out to a C function, we *do* leave enough space in the stack frame for callee- saved registers, right?
I concede to being a bit rusty here... :)
On Thu, 2015-03-26 at 09:50 +0000, David Woodhouse wrote:
On Thu, 2015-03-26 at 08:57 +0000, Richard W.M. Jones wrote:
Is pthread_create used even for the main thread in the process (I thought the kernel created that). In any case threads aren't being used explicitly by this process.
Anyway I guess the guard page is just a hole that causes the process to abort (as observed) and is not a mechanism for automatically growing the stack.
Right, It's a mechanism for catching stack overflow reliably, rather than letting it cause random memory corruption.
But for the original stack created by the kernel when the process is created, I think you just don't have the guard page. All that stops you is the ulimit.
As David mentions the original stack is created by the kernel (along with the argv, env, and auxv for process) in high virtual.
3fff8cd30000-3fff8cd50000 r-xp 00000000 00:00 0 [vdso] 3fff8cd50000-3fff8cd80000 r-xp 00000000 08:00 12456160 /lib/powerpc64le-linux-gnu/ld-2.19.so 3fff8cd80000-3fff8cd90000 rw-p 00020000 08:02 12456160 /lib/powerpc64le-linux-gnu/ld-2.19.so 3fffd0df0000-3fffd0e20000 rw-p 00000000 00:00 0 [stack]
Changing the ulimit (-s) effects the space left between the top of the address space and where the kernel loads the dynamic linker (ld.so). For example with unlimit -s 16384:
3fffae380000-3fffae3a0000 r-xp 00000000 00:00 0 [vdso] 3fffae3a0000-3fffae3d0000 r-xp 00000000 08:02 12456160 /lib/powerpc64le-linux-gnu/ld-2.19.so 3fffae3d0000-3fffae3e0000 rw-p 00020000 08:02 12456160 /lib/powerpc64le-linux-gnu/ld-2.19.so 3ffff1a10000-3ffff1a40000 rw-p 00000000 00:00 0 [stack]
It also effects the allocation of thread stacks (GLIBC uses getrlimit() then uses that value to allocate the thread stacks.
3fffaa160000-3fffab150000 rw-p 00000000 00:00 0 [stack: 29163] 3fffab150000-3fffab160000 ---p 00000000 00:00 0 3fffab160000-3fffac150000 rw-p 00000000 00:00 0 3fffac150000-3fffac160000 ---p 00000000 00:00 0 3fffac160000-3fffad150000 rw-p 00000000 00:00 0 3fffad150000-3fffad160000 ---p 00000000 00:00 0 3fffad160000-3fffae150000 rw-p 00000000 00:00 0
So... we could potentially look for ways to reduce the stack usage of OCaml-compiled code, or you get to keep increasing the limit.
It might be worth taking a look at the stack frames and seeing if I did anything entirely stupid when setting them up, and whether they can be made smaller. Although if you said it was averaging ~168 bytes, that doesn't seem *particularly* excessive. Can you see how big the stack actually does get in your worst case, and how much we'd have to shrink by in order to get away without tweaking the ulimit?
Perhaps we could also look at using a smaller stack frame between OCaml functions. If we're never using certain registers in the OCaml code generator, we don't need to save them or leave space for them on the stack. All we need to do it ensure that if we're calling out to a C function, we *do* leave enough space in the stack frame for callee- saved registers, right?
I concede to being a bit rusty here... :)
-
David Woodhouse -
Richard W.M. Jones -
Steven Munroe