Some old mails that should have been sent to the list
-------- Original Message --------
Subject: Re: /bpfs
Date: Wed, 1 Feb 2012 11:04:57 -0800
From: John Hawkes <jhawkes(a)penguincomputing.com>
To: Adam Young <ayoung(a)redhat.com>
I figured it out. Yes, an explicit mount is required. It doesn't
look like this explicit mount occurs in RHEL 4&5, but ... I press
onward, and I'll figure that out later. So now I've got the /bpfs to
seemingly work correctly on the RHEL 6 slave.
Now I'll work on why my simple bproc_move() program is segfaulting.
It does execute in usermode -- it can perform a syscall(__NR_bproc) to
read the bproc version number, but a syslog() causes a segfault. If I
eliminate the syslog(), then a call to bproc_currnode() works. And if
I add a printf() at the end, it prints the return value from
bproc_currnode(). Alas, running this a 2nd time causes a segfault.
I'm trying to puzzle out why it works sometimes and fails other times.
John
On Wed, Feb 1, 2012 at 10:42 AM, Adam Young<ayoung(a)redhat.com> wrote:
I really don't remember much about bpfs at all, except for the
feeling that
is was a mistake, and we should have been using sysfs instead. I was half
convinced that we should roll it back to the way it was in the bproc code
prior to moving to the 2.6 Linux Kernel, but thought it might be easier to
move forward than to move back.
As I recall, an explicit mount was required, but that is really faded in my
memory.
On 01/31/2012 09:12 PM, John Hawkes wrote:
>
> I change the /bpfs to behave a bit more like /proc, in that after the
> register_filesystem(), the bpfs code also does a kern_mount_data(),
> which does the mount, which gets into bpfs_get_sb() and then (because
> the MS_KERNMOUNT flag is set) does the bpfs_fill_super(). On the
> master node, /etc/init.d/beowulf does an explicit mount, which I think
> gets into bpfs_get_sb() a 2nd time, but now the MS_KERNMOUNT flag is
> off.
>
> On the slave, the the same MS_KERNMOUNT flag is seen by bpfs_get_sb(),
> and the same bpfs_fill_super(), and no 2nd mount is seen (at least not
> until the /etc/beowulf/fstab may do it, which would be unnecessary).
>
> But the readdir() still looks bad on the slave.
>
> On the master:
>
> readdir('/bpfs') ino:1 name:'.'
> readdir('/bpfs') ino:1 name:'..'
> readdir('/bpfs') ino:8 name:'self'
> readdir('/bpfs') ino:13 name:'-1'
> readdir('/bpfs') ino:11 name:'status'
>
> which is what I expect.
> But on the slave (after I've registered various nodes):
>
> readdir('/bpfs') ino:4018 name:.
> readdir('/bpfs') ino:3982 name:..
> stat('/bpfs/0') fails:2(No such file or directory)
> stat('/bpfs/-1') fails:2(No such file or directory)
> stat('/bpfs/self') fails:2(No such file or directory)
>
> So I've still got a big problem to solve.
>
> john
>
>
> On Tue, Jan 31, 2012 at 4:15 PM, John Hawkes
> <jhawkes(a)penguincomputing.com> wrote:
>>
>> So I think what the slave problem is ... is that nothing does an
>> actual mount of /bpfs. The /bpfs filesystem gets registered, but
>> nothing triggers a call to get the superblock or (because of that) to
>> fill the superblock.
>>
>> I see that the /proc filesystem calls kern_mount_data(). Perhaps the
>> slave side needs to do that. On the master side, the
>> /etc/init.d/beowulf script issues a mount command.
>>
>> john
>>
>> I rewrote much of the /bpfs code for RHEL 6.2, by the way, because
>> there were lots of changes in the vfs layer.
>>
>> On Tue, Jan 31, 2012 at 3:53 PM, John Hawkes
>> <jhawkes(a)penguincomputing.com> wrote:
>>>
>>> Apparently I have problems with my port of the /bpfs code, too.
>>>
>>> Things seem to be working pretty well for the master, but on the slave
>>> the behavior is strange. I discovered this while trying to understand
>>> why /bpfs/self wasn't being seen on the slave. I've got the
>>> kmod-bproc bpfs code instrumented to be verbose, and I do see the
>>> various /bpfs/ namespace entries being created, but they aren't being
>>> seen by user-level code.
>>>
>>> So I added a few lines to bpmaster and to bpslave, at the appropriate
>>> spots after the /bpfs was set up, to do:
>>> opendir("/rootfs")
>>> then readdir()
>>> and these both succeed on master and slave (although I haven't looked
>>> at what gets returned by readdir). However, on the master, I see a
>>> call to bpfs_root_readdir(), as expected, but I do *not* see this on
>>> the slave, even though both master and slave bpfs code use the same
>>> struct and the same inode and file operations.
>>>
>>> As you might expect, after the readdir() I do:
>>> stat("/bpfs/-1")
>>> etc. on the master, and
>>> stat("/bpfs/-1")
>>> on the slave... the master sees a successful stat() of all the names
>>> that I expect to be there, but on the slave the stat() calls fail.
>>> And nothing in the bpfs code seems to be called. So I conclude that
>>> the /bpfs on the master gets set up as I expect, but on the slave it
>>> doesn't get set up correctly... even though I'm doing the same
>>> operations on both for the /bpfs root and the superblock.
>>>
>>> Weird.
>>>
>>> I hope this isn't part of something subtle with the /rootfs being used
>>> as the temporary root during the initial phases of the slave boot.
>>>
>>> John