On Mon, Jun 21, 2021 at 6:01 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
> On 21/06/21 8:42 am, Kairui Song wrote:
> > Hi, Hari
> >
> > On Fri, Jun 18, 2021 at 11:31 PM Hari Bathini <hbathini(a)linux.ibm.com>
wrote:
> >> On 17/06/21 8:09 pm, Kairui Song wrote:
> >>> On Thu, Jun 17, 2021 at 10:08 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
> >>>> On 16/06/21 1:26 pm, Kairui Song wrote:
> >>>>> On Wed, Jun 16, 2021 at 2:32 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
> >>>>>> On 16/06/21 12:42 am, Kairui Song wrote:
> >>>>>>> On Tue, Jun 15, 2021 at 2:36 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 14/06/21 3:59 pm, Hari Bathini wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 10/06/21 8:29 pm, Hari Bathini wrote:
> >>>>>>>>>> In case of fadump, the initramfs image has
to be built to boot into
> >>>>>>>>>> the production environment as well as to
offload the active crash dump
> >>>>>>>>>> to the specified dump target (for boot
after crash). As the same image
> >>>>>>>>>> would be used for both boot scenarios, it
could not be built optimally
> >>>>>>>>>> while accommodating both cases.
> >>>>>>>>>>
> >>>>>>>>>> Use --include to include the initramfs
image built for offloading
> >>>>>>>>>> active crash dump to the specified dump
target. Also, introduce a new
> >>>>>>>>>> out-of-tree dracut module (99zz-fadumpinit)
that installs a customized
> >>>>>>>>>> init program while moving the default /init
to /init.dracut. This
> >>>>>>>>>> customized init program is leveraged to
isolate fadump image within
> >>>>>>>>>> the default initramfs image by kicking off
default boot process
> >>>>>>>>>> (exec /init.dracut) for regular boot
scenario and activating fadump
> >>>>>>>>>> initramfs image, if the system is booting
after a crash.
> >>>>>>>>>>
> >>>>>>>>>> Signed-off-by: Hari Bathini
<hbathini(a)linux.ibm.com>
> >>>>>>>>>> ---
> >>>>>>>>>>
> >>>>>>>>>> * Applying this patch, the initramfs image
built with fadump support
> >>>>>>>>>> can be too large, without
> >>>>>>>>>>
https://github.com/dracutdevs/dracut/pull/1532
> >>>>>>>>>>
> >>>>>>>>>> TODO:
> >>>>>>>>>> * Fix missing ssh identify file
/root/.ssh/kdump_id_rsa with
> >>>>>>>>>> squash not used and remote (ssh)
dump target configured.
> >>>>>>>>>> Changes in V3:
> >>>>>>>>>> * Removed dependency with squash module by
extracting fadump
> >>>>>>>>>> initramfs to /fadumproot directory.
> >>>>>>>>>> * Moved code that builds initramfs for
fadump to mkfadumprd and
> >>>>>>>>>> added cleanup support based on
inputs from Kairui.
> >>>>>>>>>>
> >>>>>>>>>> Changes in V2:
> >>>>>>>>>> * Moved kernel modules installation from
install() to installkernel() for
> >>>>>>>>>> zz-fadumpinit dracut module.
> >>>>>>>>>> * Updated kexec-tools.spec to install
zz-fadumpinit module files
> >>>>>>>>>> appropriately.
> >>>>>>>>>> * Fixed error check for capture initramfs
image isolation with dracut.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> dracut-fadump-init-fadump.sh | 20
++++++++++++++
> >>>>>>>>>> dracut-fadump-module-setup.sh | 17
++++++++++++
> >>>>>>>>>> dracut-module-setup.sh | 6
+---
> >>>>>>>>>> kdumpctl | 22
+++-------------
> >>>>>>>>>> kexec-tools.spec | 14
++++++++++
> >>>>>>>>>> mkfadumprd | 58
> >>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++
> >>>>>>>>>> 6 files changed, 115 insertions(+),
22 deletions(-)
> >>>>>>>>>> create mode 100644
dracut-fadump-init-fadump.sh
> >>>>>>>>>> create mode 100644
dracut-fadump-module-setup.sh
> >>>>>>>>>> create mode 100644 mkfadumprd
> >>>>>>>>>>
> >>>>>>>>>> diff --git a/dracut-fadump-init-fadump.sh
b/dracut-fadump-init-fadump.sh
> >>>>>>>>>> new file mode 100644
> >>>>>>>>>> index 0000000..736a7dd
> >>>>>>>>>> --- /dev/null
> >>>>>>>>>> +++ b/dracut-fadump-init-fadump.sh
> >>>>>>>>>> @@ -0,0 +1,20 @@
> >>>>>>>>>> +#!/bin/sh
> >>>>>>>>>> +PATH=/bin:/sbin
> >>>>>>>>>> +
> >>>>>>>>>> +[ -e /proc/self/mounts ] \
> >>>>>>>>>> + || (mkdir -p /proc && mount -t
proc -o nosuid,noexec,nodev proc
> >>>>>>>>>> /proc)
> >>>>>>>>>> +
> >>>>>>>>>> +grep -q '^sysfs /sys sysfs'
/proc/self/mounts \
> >>>>>>>>>> + || (mkdir -p /sys && mount -t
sysfs -o nosuid,noexec,nodev sysfs
> >>>>>>>>>> /sys)
> >>>>>>>>>> +
> >>>>>>>>>> +# If system is booting after a crash,
switch to initramfs built for
> >>>>>>>>>> dump capture
> >>>>>>>>>> +if [ -f
/proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
> >>>>>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot
]; then
> >>>>>>>>>> + mkdir -p /newroot
> >>>>>>>>>> + mount -t ramfs ramfs /newroot
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Hi Kairui,
> >>>>>>>>
> >>>>>>>>>> + mv /fadumproot/* /newroot
> >>>>>>>>
> >>>>>>>>> Similar to the dracut --include issue, this
would also leave behind
> >>>>>>>>> filenames beginning with '.'. As
replacing '*' with '.' won't work,
> >>>>>>>>
> >>>>>>>> How about the below instead:
> >>>>>>>>
> >>>>>>>> for FILE in $(ls -A fadumproot); do mv
fadumproot/$FILE newroot; done
> >>>>>>>>
> >>>>>>>> Need to install 'ls' while setting up the
module though..
> >>>>>>>>
> >>>>>>>
> >>>>>>> Hi Hari,
> >>>>>>
> >>>>>> Hi Kairui,
> >>>>>>
> >>>>>>>
> >>>>>>> After testing with squash module enabled for both
initramfs and the
> >>>>>>> embedded fadump initramfs, I think your previous idea
that use `cp`
> >>>>>>> consumes same amount of memory as with `mv` is right.
> >>>>>>> Using `mv` will only save memory when the normal
initramfs is not
> >>>>>>> using squash module, but I believe we will have to
enable squash
> >>>>>>> module for both initramfs to save memory for fadump, am
I right?
> >>>>>>
> >>>>>> It would be better if we can stick with the same build
parameters for
> >>>>>> the default initramfs except for "--add
zz-fadumpinit". So, the way
> >>>>>> I look at it, using squash module to build default
initramfs image
> >>>>>> must depend on whether it was already with squash or not.
Let me know
> >>>>>> your thoughts..
> >>>>>
> >>>>> I recommended both initramfs to use squash image, this will
help to
> >>>>> minimize the memory usage for fadump, so we can use same
crashkernel=
> >>>>> value for both fadump and kdump.
> >>>>> In my test, when both initramfs uses squash module, fadump
works with
> >>>>> 384M of memory, Without squash module enabled for both
initramfs,
> >>>>> seems have to reserve 512M of memory. No error or warning
message
> >>>>> observed. (just a umount failure, which happens once in the
umount
> >>>>> loop in init script, that's expected, we can silence that
later)
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> And base on that assumption (both initramfs uses squash
module), I
> >>>>>>> made a scratch `init` file for fadump setup, this one
should work fine
> >>>>>>> and consumes less memory, simply call `switch_root`
will left the old
> >>>>>>> mount points uncleared so the old squash img is never
dropped and
> >>>>>>> consumes ~100m of memory. How do you think? Other boot
init program
> >>>>>>> will cleanup old mount points recursively (eg.
systemd), so I think it
> >>>>>>> make sense for this script to do something similar.
> >>>>>>> (this require 99squash module to install umount as
well, and
> >>>>>>> 99zz-fadump will need to install pivot_root)
> >>>>>>
> >>>>>> I would like to test this. Other than installing umount
with squash
> >>>>>> module, any other changes needed (say in init-squash.sh)?
> >>>>>
> >>>>
> >>>> Hi Kairui,
> >>>>
> >>>>> Just install `umount` and include the previous dracut pull
requests we
> >>>>> opened, nothing else needed. I'm also testing this on
fadump capable
> >>>>> machine, so far everything looks good.
> >>>>>
> >>>>
> >>>> I was trying to test it this afternoon. Ran into OOM issues with
the
> >>>> above patch in capture kernel..
> >>>>
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>> #!/bin/sh
> >>>>>>> export PATH=/bin:/sbin
> >>>>>>>
> >>>>>>> [ -e /proc/self/mounts ] \
> >>>>>>> || (mkdir -p /proc && mount -t proc -o
nosuid,noexec,nodev proc /proc)
> >>>>>>>
> >>>>>>> grep -q '^sysfs /sys sysfs' /proc/self/mounts
\
> >>>>>>> || (mkdir -p /sys && mount -t sysfs -o
nosuid,noexec,nodev sysfs /sys)
> >>>>>>>
> >>>>>>> if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [
-f
> >>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
> >>>>>>> mkdir /newroot
> >>>>>>> mount -t ramfs ramfs /newroot
> >>>>>>>
> >>>>>>> cp --reflink=auto --sparse=auto
--preserve=mode,timestamps,links
> >>>>>>> -dfr /fadumproot/. /newroot/
> >>>>>>>
> >>>>>>> mkdir /newroot/sys
> >>>>>>> mkdir /newroot/proc
> >>>>>>> mkdir /newroot/oldroot
> >>>>>>> mount --move /sys /newroot/sys
> >>>>>>> mount --move /proc /newroot/proc
> >>>>>>>
> >>>>>>> cd /newroot || exit 1
> >>>>>>> pivot_root . oldroot
> >>>>>>>
> >>>>>>> # Clean up all mount points in oldroot
> >>>>>>> while true; do
> >>>>>>> loop=0
> >>>>>>> while read -r _ mp _; do
> >>>>>>> case $mp in
> >>>>>>> /oldroot/* )
> >>>>>>> umount "$mp"
&& loop=1
> >>>>>>> ;;
> >>>>>>> esac
> >>>>>>> done < /proc/mounts
> >>>>>>> [ $loop = 1 ] || break
> >>>>>>> done
> >>>>>>> umount -l oldroot
> >>>>>>>
> >>>>
> >>>>>>> # Jump to initramfs built for dump capture
> >>>>>>> exec /init
> >>>>
> >>>> Must have to do with the above. At the least, "exec
/newroot/init".
> >>>> Also, isn't this missing a chroot..
> >>>
> >>> Hi Hari,
> >>>
> >>> Did you miss any pull request from dracut? Like:
> >>>
https://github.com/dracutdevs/dracut/pull/1532
> >> >
> >> Sorry, I could have been clearer. When squash module is used on both
> >> default and fadump initramfs, it is working but when the default
> >> initramfs is built without squash enabled, running into the OOM
> >> issues in capture kernel. Did you get a chance to try this scenario?
> >> If yes, no issues in this scenario as well? fwiw, I see the below
> >> trace in capture kernel when default initramfs is not built with
> >> squash module but only "--add zz-fadumpinit":
> >>
> >> [ 1.191970] Freeing unused kernel memory: 6528K
> >> [ 1.191978] This architecture does not have kernel memory
protection.
> >> [ 1.191988] Run /init as init process
> >> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
> >> /init: line 44: /proc/mounts: No such file or directory
> >> umount: oldroot: umount failed: No such file or directory.
> >> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
> >> /init: line 44: /proc/mounts: No such file or directory
> >> umount: oldroot: umount failed: No such file or directory.
> >> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
> >> /init: line 44: /proc/mounts: No such file or directory
> >> umount: oldroot: umount failed: No such file or directory.
> >> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
> >> /init: line 44: /proc/mounts: No such file or directory
> >> umount: oldroot: umount failed: No such file or directory.
> >> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
> >> /init: line 44: /proc/mounts: No such file or directory
> >> umount: oldroot: umount failed: No such file or directory.
> >> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
> >> /init: line 44: /proc/mounts: No such file or directory
> >> umount: oldroot: umount failed: No such file or directory.
> >> [ 1.389399] cp invoked oom-killer:
gfp_mask=0x100cc2(GFP_HIGHUSER),
> >> order=0, oom_score_adj=0
> >> [ 1.389435] CPU: 6 PID: 424 Comm: cp Not tainted
> >> 5.12.10-300.fc34.ppc64le #1
> >> [ 1.389443] Call Trace:
> >> [ 1.389446] [c0000000199df310] [c0000000009e881c]
> >> dump_stack+0xc0/0x104 (unreliable)
> >> [ 1.389460] [c0000000199df360] [c0000000003d12a8]
dump_header+0x64/0x284
> >> [ 1.389469] [c0000000199df3f0] [c0000000003cf970]
> >> oom_kill_process+0x2c0/0x2d0
> >> [ 1.389477] [c0000000199df430] [c0000000003d0c14]
> >> out_of_memory+0x224/0x780
> >> [ 1.389485] [c0000000199df4d0] [c00000000045c7f0]
> >> __alloc_pages_slowpath.constprop.0+0xc60/0xda0
> >> [ 1.389494] [c0000000199df680] [c00000000045cc50]
> >> __alloc_pages_nodemask+0x320/0x3d0
> >> [ 1.389503] [c0000000199df710] [c00000000048b95c]
> >> alloc_pages_current+0xcc/0x190
> >> [ 1.389511] [c0000000199df760] [c0000000003c677c]
> >> pagecache_get_page+0x36c/0x840
> >> [ 1.389520] [c0000000199df840] [c0000000003c7454]
> >> grab_cache_page_write_begin+0x34/0x70
> >> [ 1.389529] [c0000000199df870] [c000000000548504]
> >> simple_write_begin+0x44/0x100
> >> [ 1.389537] [c0000000199df8b0] [c0000000003c17fc]
> >> generic_perform_write+0xec/0x260
> >> [ 1.389546] [c0000000199df950] [c0000000003c8d70]
> >> __generic_file_write_iter+0x180/0x2f0
> >> [ 1.389555] [c0000000199df9b0] [c0000000003c8f8c]
> >> generic_file_write_iter+0xac/0x110
> >> [ 1.389563] [c0000000199df9f0] [c0000000004face0]
> >> do_iter_readv_writev+0x200/0x250
> >> [ 1.389572] [c0000000199dfa50] [c0000000004fbf78]
> >> do_iter_write+0xc8/0x270
> >> [ 1.389579] [c0000000199dfaa0] [c00000000055aa68]
> >> iter_file_splice_write+0x238/0x500
> >> [ 1.389587] [c0000000199dfbb0] [c000000000559fac]
> >> direct_splice_actor+0x4c/0x80
> >> [ 1.389595] [c0000000199dfbd0] [c0000000005598e0]
> >> splice_direct_to_actor+0x110/0x320
> >> [ 1.389603] [c0000000199dfc50] [c000000000559b98]
> >> do_splice_direct+0xa8/0x110
> >> [ 1.389611] [c0000000199dfcc0] [c0000000004fe76c]
> >> vfs_copy_file_range+0x1bc/0x5c0
> >> [ 1.389619] [c0000000199dfd40] [c0000000004fec5c]
> >> sys_copy_file_range+0xec/0x270
> >> [ 1.389626] [c0000000199dfdb0] [c00000000002ddfc]
> >> system_call_exception+0x10c/0x270
> >> [ 1.389635] [c0000000199dfe10] [c00000000000d45c]
> >> system_call_common+0xec/0x278
> >> [ 1.389643] --- interrupt: c00 at 0x7fffa0ff6bc8
> >>
> >>
> >> Thanks
> >> Hari
> >>
>
> Hi Kairui
>
> > Yes, I did ran into OOM after a panic, when the normal inistramfs is
> > not built with squash module enbled. And this happens with every
> > version of the patch, that's why I recommended to enabled squash
> > module in normal initramfs so fadump can have a much smaller memory
>
> With V3, I don't see OOM when the default initramfs is not built with
> squash module. I guess, the case you are referring to is in comparison
> to squash enabled vs squash not enabled. I think the additional memory
> requirement for no squash case is because both initramfs images are
> unpacked for a while before switch_root can clean up the old
> initramfs/root. So, that memory usage will go down after switch_root
> cleans up the oldroot. So, a slight increase in the memory reserved for
> fadump in the no squash enabled case is likely to resolve the OOM
> you are referring to.
Hi Hari
The new scratch fadump-init I sent previously, does not support normal
initramfs without squash module, I've made a new fadump-init that is
compatible with both squash-enabled and non-squash-enabled normal
initramfs.
>
> But with pivot_root approach, no squash default initramfs case is
> likely to fail even after increasing memory as the same /init script
> is executing multiple times. I spiked the memory reserved for fadump
> to 4096MB from 512MB, the log is longer with multiple pivot_root
> instance errors but it eventually hits OOM even with 4096MB reserved
> for it (refer attachment)...
With the new init in the attachment, I think it will work just like
before for non-squash case.
>
> > usage. I think it should be safe to do so (enabled squash module), so
> > I think force enabled squash module when fadump is enabled is
> > acceptable, how do you think?
>
> I am fine with squash being used for default initramfs. But I would want
> the case where squash is not enabled for default initramfs also to be
> supported though. Behavior above makes me skeptical whether we can
> support both cases seamlessly with the pivot_root approach proposed.
>
> > Also the new scratch init only works when squash module is enabled
> > since it uses pivot_root, we can make it detect if rootfs is a mount
> > point and change to use switch_root if we want to be compatible with
> > normal initramfs built without squash module enabled.
>
> Right, pivot_root approach has issues when default initramfs is not
> built with squash.
>
> How about replacing "mv" with "cp" in V3 and build default
initramfs
> without squash by default for now and fix switch_root for squash case
> (do cleanup and umount overlay) eventually and enable squash by
> default on default initramfs after that switch_root change?
The reason I tried to use pivot_root for squash case is that, when
squash module is enabled for normal initramfs, rootfs is not ramfs,
but a overlay mount of a squashfs. So someone has to do a recursive
umount of the rootfs, or the content of rootfs is never properly
cleared. switch_root will just try to `rm` all files in the original
root, but won't do any umount work... So the loop device used by
squash module and other resources are unmanaged and leak during the
whole dump progress, which wastes about 100M (size of the normal
initramfs).
And if we don't enable squash module for normal initramfs, fadump
still consumes slightly more memory than kdump (roughly 512M vs 384M).
It turns out not hard to make the fadump-init compatible with both
squash and non-squash case, I tested the new fadump-init with both
case and now it seems working very well. I don't have a ppc machine
available now so I just tested on VM emulating a fadump. I've put the
patch in the attachment, please have a try and let me know if anything
is not working.
And I've opened a new pull request to dracut, it's needed if normal
initramfs is built with squash module enabled:
https://github.com/dracutdevs/dracut/pull/1545
And previous PRs are also needed:
https://github.com/dracutdevs/dracut/pull/1535
https://github.com/dracutdevs/dracut/pull/1532
>
> But if you insist, I am ok with pivot_root approach as long as default
> initramfs not having squash enabled case works alright..
>
> Thanks
> Hari
--
Best Regards,
Kairui Song
Hi Hari, I found this scratch patch still have some flaw, and here is
a updated version, tested with fadump machines:
--
Best Regards,
Kairui Song