This change is probably relevant to how BProc does the system call.
-------- Original Message --------
Subject: Re: Scyld filecache, /rootfs, etc.
Date: Mon, 23 Jan 2012 23:01:49 -0500
From: Adam Young <ayoung(a)redhat.com>
To: John Hawkes <jhawkes(a)penguincomputing.com>
On 01/23/2012 03:19 PM, John Hawkes wrote:
On Mon, Jan 23, 2012 at 11:59 AM, John Hawkes
<jhawkes(a)penguincomputing.com> wrote:
>> If they are loaded before the migrate, they get there via task_packer. If
>> not, they need to be in place. I forget the order that things happen
>> ...been too long. You can use an nfs root to preposition the libraries.
> Ohhh right right right. It's "bpsh -M" to do an exec-move, vs. the
> default move-exec.
> I tried that, and I see the same quick-exit.
Ummm no, "bpsh -M" is the default. It may do the exec locally on the
master, but that doesn't run the loader to load the libraries. That
is done on the compute node.
When I do "bpsh -m" -- MOVE-EXEC -- then it doesn't try to load the
libraries, but it does fail quickly, and this time with an exit code
of 11 (0xb), after it exits to user space. I'm thinking maybe I have
the bproc_kernel_thread and bproc_child_rip wrong. I already had to
fix the bproc_kernel_thread for RHEL6. Maybe I did that wrong, or
maybe I need to make changes to bproc_child_rip, too. One diff
between the RHEL5 and RHEL6 kernel_thread is the addition of an extra
push:
/usr/src/debug////////kernel-2.6.32-220.2.1.el6/linux-2.6.32-220.2.1.el6.620a0001.x86_64/arch/x86/kernel/entry_64.S:1172
ffffffff8100c060<kernel_thread> 31 c0 xor %eax,%eax
ffffffff8100c062<kernel_thread+0x2> 6a 18 pushq
$0x18<<<<<<<<<<<<<<<<<<<<
ffffffff8100c064<kernel_thread+0x4> 50 push %rax
ffffffff8100c065<kernel_thread+0x5> 68 00 02 00 00 pushq $0x200
ffffffff8100c06a<kernel_thread+0xa> 6a 10 pushq $0x10
ffffffff8100c06c<kernel_thread+0xc> 68 40 c1 00 81 pushq
$0xffffffff8100c140
ffffffff8100c071<kernel_thread+0x11> 50 push %rax
John
I don't have the RHEL5 code handy. I am not sure exactly which line is
the "extra push"
The comment is /* push in order ss, rsp, eflags, cs, rip */
Line 1172 is the macro FAKE_STACK_FRAME $child_rip
From the git blame of Linus' tree:
1cf8343f arch/x86/kernel/entry_64.S (Seiichi Ikarashi
2011-12-06 17:58:14 +0900 225) pushq_cfi
$(X86_EFLAGS_IF|X86_EFLAGS_BIT1) /* eflags - interrupts on */
x86: Fix rflags in FAKE_STACK_FRAME
The x86_64 kernel pushes the fake kernel stack in
arch/x86/kernel/entry_64.S:FAKE_STACK_FRAME, and
rflags register in it does not conform to the specification.
Although Intel's manual[1] says bit 1 of it shall be set to 1,
this bit is cleared to 0 on pushing the fake stack.
[1] Intel(R) 64 and IA-32 Architectures Software Developer's Manual
Vol.1 3-21 Figure 3-8. EFLAGS Register
If it is not on purpose, it is better to be fixed, because
it can lead some tools misunderstanding the stack frame. For example,
"crash" utility[2] actually detects it and warns you like
below:
But I suspect that was just the change:
- pushq_cfi $X86_EFLAGS_IF /* eflags - interrupts on */
+ pushq_cfi $(X86_EFLAGS_IF|X86_EFLAGS_BIT1) /* eflags -
interrupts on */
Before that it was:
df5d1874 arch/x86/kernel/entry_64.S (Jan Beulich
2010-09-02 14:07:16 +0100 224) pushq_cfi $X86_EFLAGS_IF /* eflags -
interrupts on */
x86: Use {push,pop}{l,q}_cfi in more places
... plus additionally introduce {push,pop}f{l,q}_cfi. All in the
hope that the code becomes better readable this way (it gets
quite a bit smaller in any case).
which I guess is where the _cfi comes from.
Which push didn't exist in the RHEL 5 Kernel?