Hi, Hari
On Fri, Jun 18, 2021 at 11:31 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
> On 17/06/21 8:09 pm, Kairui Song wrote:
>> On Thu, Jun 17, 2021 at 10:08 PM Hari Bathini <hbathini(a)linux.ibm.com>
wrote:
>>> On 16/06/21 1:26 pm, Kairui Song wrote:
>>>> On Wed, Jun 16, 2021 at 2:32 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>> On 16/06/21 12:42 am, Kairui Song wrote:
>>>>>> On Tue, Jun 15, 2021 at 2:36 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 14/06/21 3:59 pm, Hari Bathini wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/06/21 8:29 pm, Hari Bathini wrote:
>>>>>>>>> In case of fadump, the initramfs image has to be
built to boot into
>>>>>>>>> the production environment as well as to offload the
active crash dump
>>>>>>>>> to the specified dump target (for boot after crash).
As the same image
>>>>>>>>> would be used for both boot scenarios, it could not
be built optimally
>>>>>>>>> while accommodating both cases.
>>>>>>>>>
>>>>>>>>> Use --include to include the initramfs image built
for offloading
>>>>>>>>> active crash dump to the specified dump target. Also,
introduce a new
>>>>>>>>> out-of-tree dracut module (99zz-fadumpinit) that
installs a customized
>>>>>>>>> init program while moving the default /init to
/init.dracut. This
>>>>>>>>> customized init program is leveraged to isolate
fadump image within
>>>>>>>>> the default initramfs image by kicking off default
boot process
>>>>>>>>> (exec /init.dracut) for regular boot scenario and
activating fadump
>>>>>>>>> initramfs image, if the system is booting after a
crash.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Hari Bathini
<hbathini(a)linux.ibm.com>
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> * Applying this patch, the initramfs image built with
fadump support
>>>>>>>>> can be too large, without
>>>>>>>>>
https://github.com/dracutdevs/dracut/pull/1532
>>>>>>>>>
>>>>>>>>> TODO:
>>>>>>>>> * Fix missing ssh identify file
/root/.ssh/kdump_id_rsa with
>>>>>>>>> squash not used and remote (ssh) dump target
configured.
>>>>>>>>> Changes in V3:
>>>>>>>>> * Removed dependency with squash module by extracting
fadump
>>>>>>>>> initramfs to /fadumproot directory.
>>>>>>>>> * Moved code that builds initramfs for fadump to
mkfadumprd and
>>>>>>>>> added cleanup support based on inputs from
Kairui.
>>>>>>>>>
>>>>>>>>> Changes in V2:
>>>>>>>>> * Moved kernel modules installation from install() to
installkernel() for
>>>>>>>>> zz-fadumpinit dracut module.
>>>>>>>>> * Updated kexec-tools.spec to install zz-fadumpinit
module files
>>>>>>>>> appropriately.
>>>>>>>>> * Fixed error check for capture initramfs image
isolation with dracut.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> dracut-fadump-init-fadump.sh | 20
++++++++++++++
>>>>>>>>> dracut-fadump-module-setup.sh | 17
++++++++++++
>>>>>>>>> dracut-module-setup.sh | 6 +---
>>>>>>>>> kdumpctl | 22
+++-------------
>>>>>>>>> kexec-tools.spec | 14
++++++++++
>>>>>>>>> mkfadumprd | 58
>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++
>>>>>>>>> 6 files changed, 115 insertions(+), 22
deletions(-)
>>>>>>>>> create mode 100644
dracut-fadump-init-fadump.sh
>>>>>>>>> create mode 100644
dracut-fadump-module-setup.sh
>>>>>>>>> create mode 100644 mkfadumprd
>>>>>>>>>
>>>>>>>>> diff --git a/dracut-fadump-init-fadump.sh
b/dracut-fadump-init-fadump.sh
>>>>>>>>> new file mode 100644
>>>>>>>>> index 0000000..736a7dd
>>>>>>>>> --- /dev/null
>>>>>>>>> +++ b/dracut-fadump-init-fadump.sh
>>>>>>>>> @@ -0,0 +1,20 @@
>>>>>>>>> +#!/bin/sh
>>>>>>>>> +PATH=/bin:/sbin
>>>>>>>>> +
>>>>>>>>> +[ -e /proc/self/mounts ] \
>>>>>>>>> + || (mkdir -p /proc && mount -t proc -o
nosuid,noexec,nodev proc
>>>>>>>>> /proc)
>>>>>>>>> +
>>>>>>>>> +grep -q '^sysfs /sys sysfs'
/proc/self/mounts \
>>>>>>>>> + || (mkdir -p /sys && mount -t sysfs -o
nosuid,noexec,nodev sysfs
>>>>>>>>> /sys)
>>>>>>>>> +
>>>>>>>>> +# If system is booting after a crash, switch to
initramfs built for
>>>>>>>>> dump capture
>>>>>>>>> +if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] ||
[ -f
>>>>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>>>>>>> + mkdir -p /newroot
>>>>>>>>> + mount -t ramfs ramfs /newroot
>>>>>>>
>>>>>>>
>>>>>>> Hi Kairui,
>>>>>>>
>>>>>>>>> + mv /fadumproot/* /newroot
>>>>>>>
>>>>>>>> Similar to the dracut --include issue, this would also
leave behind
>>>>>>>> filenames beginning with '.'. As replacing
'*' with '.' won't work,
>>>>>>>
>>>>>>> How about the below instead:
>>>>>>>
>>>>>>> for FILE in $(ls -A fadumproot); do mv
fadumproot/$FILE newroot; done
>>>>>>>
>>>>>>> Need to install 'ls' while setting up the module
though..
>>>>>>>
>>>>>>
>>>>>> Hi Hari,
>>>>>
>>>>> Hi Kairui,
>>>>>
>>>>>>
>>>>>> After testing with squash module enabled for both initramfs and
the
>>>>>> embedded fadump initramfs, I think your previous idea that use
`cp`
>>>>>> consumes same amount of memory as with `mv` is right.
>>>>>> Using `mv` will only save memory when the normal initramfs is
not
>>>>>> using squash module, but I believe we will have to enable squash
>>>>>> module for both initramfs to save memory for fadump, am I right?
>>>>>
>>>>> It would be better if we can stick with the same build parameters
for
>>>>> the default initramfs except for "--add zz-fadumpinit". So,
the way
>>>>> I look at it, using squash module to build default initramfs image
>>>>> must depend on whether it was already with squash or not. Let me
know
>>>>> your thoughts..
>>>>
>>>> I recommended both initramfs to use squash image, this will help to
>>>> minimize the memory usage for fadump, so we can use same crashkernel=
>>>> value for both fadump and kdump.
>>>> In my test, when both initramfs uses squash module, fadump works with
>>>> 384M of memory, Without squash module enabled for both initramfs,
>>>> seems have to reserve 512M of memory. No error or warning message
>>>> observed. (just a umount failure, which happens once in the umount
>>>> loop in init script, that's expected, we can silence that later)
>>>>
>>>>>
>>>>>>
>>>>>> And base on that assumption (both initramfs uses squash module),
I
>>>>>> made a scratch `init` file for fadump setup, this one should work
fine
>>>>>> and consumes less memory, simply call `switch_root` will left the
old
>>>>>> mount points uncleared so the old squash img is never dropped
and
>>>>>> consumes ~100m of memory. How do you think? Other boot init
program
>>>>>> will cleanup old mount points recursively (eg. systemd), so I
think it
>>>>>> make sense for this script to do something similar.
>>>>>> (this require 99squash module to install umount as well, and
>>>>>> 99zz-fadump will need to install pivot_root)
>>>>>
>>>>> I would like to test this. Other than installing umount with squash
>>>>> module, any other changes needed (say in init-squash.sh)?
>>>>
>>>
>>> Hi Kairui,
>>>
>>>> Just install `umount` and include the previous dracut pull requests we
>>>> opened, nothing else needed. I'm also testing this on fadump capable
>>>> machine, so far everything looks good.
>>>>
>>>
>>> I was trying to test it this afternoon. Ran into OOM issues with the
>>> above patch in capture kernel..
>>>
>>>>>
>>>>>
>>>>>>
>>>>>> #!/bin/sh
>>>>>> export PATH=/bin:/sbin
>>>>>>
>>>>>> [ -e /proc/self/mounts ] \
>>>>>> || (mkdir -p /proc && mount -t proc -o
nosuid,noexec,nodev proc /proc)
>>>>>>
>>>>>> grep -q '^sysfs /sys sysfs' /proc/self/mounts \
>>>>>> || (mkdir -p /sys && mount -t sysfs -o
nosuid,noexec,nodev sysfs /sys)
>>>>>>
>>>>>> if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>>>> mkdir /newroot
>>>>>> mount -t ramfs ramfs /newroot
>>>>>>
>>>>>> cp --reflink=auto --sparse=auto
--preserve=mode,timestamps,links
>>>>>> -dfr /fadumproot/. /newroot/
>>>>>>
>>>>>> mkdir /newroot/sys
>>>>>> mkdir /newroot/proc
>>>>>> mkdir /newroot/oldroot
>>>>>> mount --move /sys /newroot/sys
>>>>>> mount --move /proc /newroot/proc
>>>>>>
>>>>>> cd /newroot || exit 1
>>>>>> pivot_root . oldroot
>>>>>>
>>>>>> # Clean up all mount points in oldroot
>>>>>> while true; do
>>>>>> loop=0
>>>>>> while read -r _ mp _; do
>>>>>> case $mp in
>>>>>> /oldroot/* )
>>>>>> umount "$mp" && loop=1
>>>>>> ;;
>>>>>> esac
>>>>>> done < /proc/mounts
>>>>>> [ $loop = 1 ] || break
>>>>>> done
>>>>>> umount -l oldroot
>>>>>>
>>>
>>>>>> # Jump to initramfs built for dump capture
>>>>>> exec /init
>>>
>>> Must have to do with the above. At the least, "exec
/newroot/init".
>>> Also, isn't this missing a chroot..
>>
>> Hi Hari,
>>
>> Did you miss any pull request from dracut? Like:
>>
https://github.com/dracutdevs/dracut/pull/1532
> >
> Sorry, I could have been clearer. When squash module is used on both
> default and fadump initramfs, it is working but when the default
> initramfs is built without squash enabled, running into the OOM
> issues in capture kernel. Did you get a chance to try this scenario?
> If yes, no issues in this scenario as well? fwiw, I see the below
> trace in capture kernel when default initramfs is not built with
> squash module but only "--add zz-fadumpinit":
>
> [ 1.191970] Freeing unused kernel memory: 6528K
> [ 1.191978] This architecture does not have kernel memory protection.
> [ 1.191988] Run /init as init process
> pivot_root: failed to change root from `.' to `oldroot': Invalid
argument
> /init: line 44: /proc/mounts: No such file or directory
> umount: oldroot: umount failed: No such file or directory.
> pivot_root: failed to change root from `.' to `oldroot': Invalid
argument
> /init: line 44: /proc/mounts: No such file or directory
> umount: oldroot: umount failed: No such file or directory.
> pivot_root: failed to change root from `.' to `oldroot': Invalid
argument
> /init: line 44: /proc/mounts: No such file or directory
> umount: oldroot: umount failed: No such file or directory.
> pivot_root: failed to change root from `.' to `oldroot': Invalid
argument
> /init: line 44: /proc/mounts: No such file or directory
> umount: oldroot: umount failed: No such file or directory.
> pivot_root: failed to change root from `.' to `oldroot': Invalid
argument
> /init: line 44: /proc/mounts: No such file or directory
> umount: oldroot: umount failed: No such file or directory.
> pivot_root: failed to change root from `.' to `oldroot': Invalid
argument
> /init: line 44: /proc/mounts: No such file or directory
> umount: oldroot: umount failed: No such file or directory.
> [ 1.389399] cp invoked oom-killer: gfp_mask=0x100cc2(GFP_HIGHUSER),
> order=0, oom_score_adj=0
> [ 1.389435] CPU: 6 PID: 424 Comm: cp Not tainted
> 5.12.10-300.fc34.ppc64le #1
> [ 1.389443] Call Trace:
> [ 1.389446] [c0000000199df310] [c0000000009e881c]
> dump_stack+0xc0/0x104 (unreliable)
> [ 1.389460] [c0000000199df360] [c0000000003d12a8] dump_header+0x64/0x284
> [ 1.389469] [c0000000199df3f0] [c0000000003cf970]
> oom_kill_process+0x2c0/0x2d0
> [ 1.389477] [c0000000199df430] [c0000000003d0c14]
> out_of_memory+0x224/0x780
> [ 1.389485] [c0000000199df4d0] [c00000000045c7f0]
> __alloc_pages_slowpath.constprop.0+0xc60/0xda0
> [ 1.389494] [c0000000199df680] [c00000000045cc50]
> __alloc_pages_nodemask+0x320/0x3d0
> [ 1.389503] [c0000000199df710] [c00000000048b95c]
> alloc_pages_current+0xcc/0x190
> [ 1.389511] [c0000000199df760] [c0000000003c677c]
> pagecache_get_page+0x36c/0x840
> [ 1.389520] [c0000000199df840] [c0000000003c7454]
> grab_cache_page_write_begin+0x34/0x70
> [ 1.389529] [c0000000199df870] [c000000000548504]
> simple_write_begin+0x44/0x100
> [ 1.389537] [c0000000199df8b0] [c0000000003c17fc]
> generic_perform_write+0xec/0x260
> [ 1.389546] [c0000000199df950] [c0000000003c8d70]
> __generic_file_write_iter+0x180/0x2f0
> [ 1.389555] [c0000000199df9b0] [c0000000003c8f8c]
> generic_file_write_iter+0xac/0x110
> [ 1.389563] [c0000000199df9f0] [c0000000004face0]
> do_iter_readv_writev+0x200/0x250
> [ 1.389572] [c0000000199dfa50] [c0000000004fbf78]
> do_iter_write+0xc8/0x270
> [ 1.389579] [c0000000199dfaa0] [c00000000055aa68]
> iter_file_splice_write+0x238/0x500
> [ 1.389587] [c0000000199dfbb0] [c000000000559fac]
> direct_splice_actor+0x4c/0x80
> [ 1.389595] [c0000000199dfbd0] [c0000000005598e0]
> splice_direct_to_actor+0x110/0x320
> [ 1.389603] [c0000000199dfc50] [c000000000559b98]
> do_splice_direct+0xa8/0x110
> [ 1.389611] [c0000000199dfcc0] [c0000000004fe76c]
> vfs_copy_file_range+0x1bc/0x5c0
> [ 1.389619] [c0000000199dfd40] [c0000000004fec5c]
> sys_copy_file_range+0xec/0x270
> [ 1.389626] [c0000000199dfdb0] [c00000000002ddfc]
> system_call_exception+0x10c/0x270
> [ 1.389635] [c0000000199dfe10] [c00000000000d45c]
> system_call_common+0xec/0x278
> [ 1.389643] --- interrupt: c00 at 0x7fffa0ff6bc8
>
>
> Thanks
> Hari
>
Yes, I did ran into OOM after a panic, when the normal inistramfs is
not built with squash module enbled. And this happens with every
version of the patch, that's why I recommended to enabled squash
module in normal initramfs so fadump can have a much smaller memory
With V3, I don't see OOM when the default initramfs is not built with
squash module. I guess, the case you are referring to is in comparison
to squash enabled vs squash not enabled. I think the additional memory
requirement for no squash case is because both initramfs images are
unpacked for a while before switch_root can clean up the old
initramfs/root. So, that memory usage will go down after switch_root
cleans up the oldroot. So, a slight increase in the memory reserved for
fadump in the no squash enabled case is likely to resolve the OOM
you are referring to.
But with pivot_root approach, no squash default initramfs case is
likely to fail even after increasing memory as the same /init script
is executing multiple times. I spiked the memory reserved for fadump
to 4096MB from 512MB, the log is longer with multiple pivot_root
instance errors but it eventually hits OOM even with 4096MB reserved
for it (refer attachment)...
usage. I think it should be safe to do so (enabled squash module),
so
I think force enabled squash module when fadump is enabled is
acceptable, how do you think?
I am fine with squash being used for default initramfs. But I would want
the case where squash is not enabled for default initramfs also to be
supported though. Behavior above makes me skeptical whether we can
support both cases seamlessly with the pivot_root approach proposed.
Also the new scratch init only works when squash module is enabled
since it uses pivot_root, we can make it detect if rootfs is a mount
point and change to use switch_root if we want to be compatible with
normal initramfs built without squash module enabled.
Right, pivot_root approach has issues when default initramfs is not
built with squash.
How about replacing "mv" with "cp" in V3 and build default initramfs
without squash by default for now and fix switch_root for squash case
(do cleanup and umount overlay) eventually and enable squash by
default on default initramfs after that switch_root change?
But if you insist, I am ok with pivot_root approach as long as default
initramfs not having squash enabled case works alright..
Thanks
Hari