On Tue, Jun 22, 2021 at 11:11 PM Kairui Song
<kasong(a)redhat.com> wrote:
>
> On Mon, Jun 21, 2021 at 6:01 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
>> On 21/06/21 8:42 am, Kairui Song wrote:
>>> Hi, Hari
>>>
>>> On Fri, Jun 18, 2021 at 11:31 PM Hari Bathini <hbathini(a)linux.ibm.com>
wrote:
>>>> On 17/06/21 8:09 pm, Kairui Song wrote:
>>>>> On Thu, Jun 17, 2021 at 10:08 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>> On 16/06/21 1:26 pm, Kairui Song wrote:
>>>>>>> On Wed, Jun 16, 2021 at 2:32 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>>>> On 16/06/21 12:42 am, Kairui Song wrote:
>>>>>>>>> On Tue, Jun 15, 2021 at 2:36 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 14/06/21 3:59 pm, Hari Bathini wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 10/06/21 8:29 pm, Hari Bathini wrote:
>>>>>>>>>>>> In case of fadump, the initramfs image
has to be built to boot into
>>>>>>>>>>>> the production environment as well as to
offload the active crash dump
>>>>>>>>>>>> to the specified dump target (for boot
after crash). As the same image
>>>>>>>>>>>> would be used for both boot scenarios, it
could not be built optimally
>>>>>>>>>>>> while accommodating both cases.
>>>>>>>>>>>>
>>>>>>>>>>>> Use --include to include the initramfs
image built for offloading
>>>>>>>>>>>> active crash dump to the specified dump
target. Also, introduce a new
>>>>>>>>>>>> out-of-tree dracut module
(99zz-fadumpinit) that installs a customized
>>>>>>>>>>>> init program while moving the default
/init to /init.dracut. This
>>>>>>>>>>>> customized init program is leveraged to
isolate fadump image within
>>>>>>>>>>>> the default initramfs image by kicking
off default boot process
>>>>>>>>>>>> (exec /init.dracut) for regular boot
scenario and activating fadump
>>>>>>>>>>>> initramfs image, if the system is booting
after a crash.
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Hari Bathini
<hbathini(a)linux.ibm.com>
>>>>>>>>>>>> ---
>>>>>>>>>>>>
>>>>>>>>>>>> * Applying this patch, the initramfs
image built with fadump support
>>>>>>>>>>>> can be too large, without
>>>>>>>>>>>>
https://github.com/dracutdevs/dracut/pull/1532
>>>>>>>>>>>>
>>>>>>>>>>>> TODO:
>>>>>>>>>>>> * Fix missing ssh identify file
/root/.ssh/kdump_id_rsa with
>>>>>>>>>>>> squash not used and remote (ssh)
dump target configured.
>>>>>>>>>>>> Changes in V3:
>>>>>>>>>>>> * Removed dependency with squash module
by extracting fadump
>>>>>>>>>>>> initramfs to /fadumproot
directory.
>>>>>>>>>>>> * Moved code that builds initramfs for
fadump to mkfadumprd and
>>>>>>>>>>>> added cleanup support based on
inputs from Kairui.
>>>>>>>>>>>>
>>>>>>>>>>>> Changes in V2:
>>>>>>>>>>>> * Moved kernel modules installation from
install() to installkernel() for
>>>>>>>>>>>> zz-fadumpinit dracut module.
>>>>>>>>>>>> * Updated kexec-tools.spec to install
zz-fadumpinit module files
>>>>>>>>>>>> appropriately.
>>>>>>>>>>>> * Fixed error check for capture initramfs
image isolation with dracut.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> dracut-fadump-init-fadump.sh |
20 ++++++++++++++
>>>>>>>>>>>> dracut-fadump-module-setup.sh |
17 ++++++++++++
>>>>>>>>>>>> dracut-module-setup.sh |
6 +---
>>>>>>>>>>>> kdumpctl |
22 +++-------------
>>>>>>>>>>>> kexec-tools.spec |
14 ++++++++++
>>>>>>>>>>>> mkfadumprd |
58
>>>>>>>>>>>>
+++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>> 6 files changed, 115
insertions(+), 22 deletions(-)
>>>>>>>>>>>> create mode 100644
dracut-fadump-init-fadump.sh
>>>>>>>>>>>> create mode 100644
dracut-fadump-module-setup.sh
>>>>>>>>>>>> create mode 100644 mkfadumprd
>>>>>>>>>>>>
>>>>>>>>>>>> diff --git a/dracut-fadump-init-fadump.sh
b/dracut-fadump-init-fadump.sh
>>>>>>>>>>>> new file mode 100644
>>>>>>>>>>>> index 0000000..736a7dd
>>>>>>>>>>>> --- /dev/null
>>>>>>>>>>>> +++ b/dracut-fadump-init-fadump.sh
>>>>>>>>>>>> @@ -0,0 +1,20 @@
>>>>>>>>>>>> +#!/bin/sh
>>>>>>>>>>>> +PATH=/bin:/sbin
>>>>>>>>>>>> +
>>>>>>>>>>>> +[ -e /proc/self/mounts ] \
>>>>>>>>>>>> + || (mkdir -p /proc && mount
-t proc -o nosuid,noexec,nodev proc
>>>>>>>>>>>> /proc)
>>>>>>>>>>>> +
>>>>>>>>>>>> +grep -q '^sysfs /sys sysfs'
/proc/self/mounts \
>>>>>>>>>>>> + || (mkdir -p /sys && mount
-t sysfs -o nosuid,noexec,nodev sysfs
>>>>>>>>>>>> /sys)
>>>>>>>>>>>> +
>>>>>>>>>>>> +# If system is booting after a crash,
switch to initramfs built for
>>>>>>>>>>>> dump capture
>>>>>>>>>>>> +if [ -f
/proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
>>>>>>>>>>>>
/proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>>>>>>>>>> + mkdir -p /newroot
>>>>>>>>>>>> + mount -t ramfs ramfs /newroot
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Kairui,
>>>>>>>>>>
>>>>>>>>>>>> + mv /fadumproot/* /newroot
>>>>>>>>>>
>>>>>>>>>>> Similar to the dracut --include issue, this
would also leave behind
>>>>>>>>>>> filenames beginning with '.'. As
replacing '*' with '.' won't work,
>>>>>>>>>>
>>>>>>>>>> How about the below instead:
>>>>>>>>>>
>>>>>>>>>> for FILE in $(ls -A fadumproot); do mv
fadumproot/$FILE newroot; done
>>>>>>>>>>
>>>>>>>>>> Need to install 'ls' while setting up the
module though..
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Hari,
>>>>>>>>
>>>>>>>> Hi Kairui,
>>>>>>>>
>>>>>>>>>
>>>>>>>>> After testing with squash module enabled for both
initramfs and the
>>>>>>>>> embedded fadump initramfs, I think your previous idea
that use `cp`
>>>>>>>>> consumes same amount of memory as with `mv` is
right.
>>>>>>>>> Using `mv` will only save memory when the normal
initramfs is not
>>>>>>>>> using squash module, but I believe we will have to
enable squash
>>>>>>>>> module for both initramfs to save memory for fadump,
am I right?
>>>>>>>>
>>>>>>>> It would be better if we can stick with the same build
parameters for
>>>>>>>> the default initramfs except for "--add
zz-fadumpinit". So, the way
>>>>>>>> I look at it, using squash module to build default
initramfs image
>>>>>>>> must depend on whether it was already with squash or not.
Let me know
>>>>>>>> your thoughts..
>>>>>>>
>>>>>>> I recommended both initramfs to use squash image, this will
help to
>>>>>>> minimize the memory usage for fadump, so we can use same
crashkernel=
>>>>>>> value for both fadump and kdump.
>>>>>>> In my test, when both initramfs uses squash module, fadump
works with
>>>>>>> 384M of memory, Without squash module enabled for both
initramfs,
>>>>>>> seems have to reserve 512M of memory. No error or warning
message
>>>>>>> observed. (just a umount failure, which happens once in the
umount
>>>>>>> loop in init script, that's expected, we can silence that
later)
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> And base on that assumption (both initramfs uses
squash module), I
>>>>>>>>> made a scratch `init` file for fadump setup, this one
should work fine
>>>>>>>>> and consumes less memory, simply call `switch_root`
will left the old
>>>>>>>>> mount points uncleared so the old squash img is never
dropped and
>>>>>>>>> consumes ~100m of memory. How do you think? Other
boot init program
>>>>>>>>> will cleanup old mount points recursively (eg.
systemd), so I think it
>>>>>>>>> make sense for this script to do something similar.
>>>>>>>>> (this require 99squash module to install umount as
well, and
>>>>>>>>> 99zz-fadump will need to install pivot_root)
>>>>>>>>
>>>>>>>> I would like to test this. Other than installing umount
with squash
>>>>>>>> module, any other changes needed (say in
init-squash.sh)?
>>>>>>>
>>>>>>
>>>>>> Hi Kairui,
>>>>>>
>>>>>>> Just install `umount` and include the previous dracut pull
requests we
>>>>>>> opened, nothing else needed. I'm also testing this on
fadump capable
>>>>>>> machine, so far everything looks good.
>>>>>>>
>>>>>>
>>>>>> I was trying to test it this afternoon. Ran into OOM issues with
the
>>>>>> above patch in capture kernel..
>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> #!/bin/sh
>>>>>>>>> export PATH=/bin:/sbin
>>>>>>>>>
>>>>>>>>> [ -e /proc/self/mounts ] \
>>>>>>>>> || (mkdir -p /proc && mount -t proc
-o nosuid,noexec,nodev proc /proc)
>>>>>>>>>
>>>>>>>>> grep -q '^sysfs /sys sysfs' /proc/self/mounts
\
>>>>>>>>> || (mkdir -p /sys && mount -t sysfs
-o nosuid,noexec,nodev sysfs /sys)
>>>>>>>>>
>>>>>>>>> if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [
-f
>>>>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>>>>>>> mkdir /newroot
>>>>>>>>> mount -t ramfs ramfs /newroot
>>>>>>>>>
>>>>>>>>> cp --reflink=auto --sparse=auto
--preserve=mode,timestamps,links
>>>>>>>>> -dfr /fadumproot/. /newroot/
>>>>>>>>>
>>>>>>>>> mkdir /newroot/sys
>>>>>>>>> mkdir /newroot/proc
>>>>>>>>> mkdir /newroot/oldroot
>>>>>>>>> mount --move /sys /newroot/sys
>>>>>>>>> mount --move /proc /newroot/proc
>>>>>>>>>
>>>>>>>>> cd /newroot || exit 1
>>>>>>>>> pivot_root . oldroot
>>>>>>>>>
>>>>>>>>> # Clean up all mount points in oldroot
>>>>>>>>> while true; do
>>>>>>>>> loop=0
>>>>>>>>> while read -r _ mp _; do
>>>>>>>>> case $mp in
>>>>>>>>> /oldroot/* )
>>>>>>>>> umount "$mp"
&& loop=1
>>>>>>>>> ;;
>>>>>>>>> esac
>>>>>>>>> done < /proc/mounts
>>>>>>>>> [ $loop = 1 ] || break
>>>>>>>>> done
>>>>>>>>> umount -l oldroot
>>>>>>>>>
>>>>>>
>>>>>>>>> # Jump to initramfs built for dump capture
>>>>>>>>> exec /init
>>>>>>
>>>>>> Must have to do with the above. At the least, "exec
/newroot/init".
>>>>>> Also, isn't this missing a chroot..
>>>>>
>>>>> Hi Hari,
>>>>>
>>>>> Did you miss any pull request from dracut? Like:
>>>>>
https://github.com/dracutdevs/dracut/pull/1532
>>>> >
>>>> Sorry, I could have been clearer. When squash module is used on both
>>>> default and fadump initramfs, it is working but when the default
>>>> initramfs is built without squash enabled, running into the OOM
>>>> issues in capture kernel. Did you get a chance to try this scenario?
>>>> If yes, no issues in this scenario as well? fwiw, I see the below
>>>> trace in capture kernel when default initramfs is not built with
>>>> squash module but only "--add zz-fadumpinit":
>>>>
>>>> [ 1.191970] Freeing unused kernel memory: 6528K
>>>> [ 1.191978] This architecture does not have kernel memory
protection.
>>>> [ 1.191988] Run /init as init process
>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>> /init: line 44: /proc/mounts: No such file or directory
>>>> umount: oldroot: umount failed: No such file or directory.
>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>> /init: line 44: /proc/mounts: No such file or directory
>>>> umount: oldroot: umount failed: No such file or directory.
>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>> /init: line 44: /proc/mounts: No such file or directory
>>>> umount: oldroot: umount failed: No such file or directory.
>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>> /init: line 44: /proc/mounts: No such file or directory
>>>> umount: oldroot: umount failed: No such file or directory.
>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>> /init: line 44: /proc/mounts: No such file or directory
>>>> umount: oldroot: umount failed: No such file or directory.
>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>> /init: line 44: /proc/mounts: No such file or directory
>>>> umount: oldroot: umount failed: No such file or directory.
>>>> [ 1.389399] cp invoked oom-killer:
gfp_mask=0x100cc2(GFP_HIGHUSER),
>>>> order=0, oom_score_adj=0
>>>> [ 1.389435] CPU: 6 PID: 424 Comm: cp Not tainted
>>>> 5.12.10-300.fc34.ppc64le #1
>>>> [ 1.389443] Call Trace:
>>>> [ 1.389446] [c0000000199df310] [c0000000009e881c]
>>>> dump_stack+0xc0/0x104 (unreliable)
>>>> [ 1.389460] [c0000000199df360] [c0000000003d12a8]
dump_header+0x64/0x284
>>>> [ 1.389469] [c0000000199df3f0] [c0000000003cf970]
>>>> oom_kill_process+0x2c0/0x2d0
>>>> [ 1.389477] [c0000000199df430] [c0000000003d0c14]
>>>> out_of_memory+0x224/0x780
>>>> [ 1.389485] [c0000000199df4d0] [c00000000045c7f0]
>>>> __alloc_pages_slowpath.constprop.0+0xc60/0xda0
>>>> [ 1.389494] [c0000000199df680] [c00000000045cc50]
>>>> __alloc_pages_nodemask+0x320/0x3d0
>>>> [ 1.389503] [c0000000199df710] [c00000000048b95c]
>>>> alloc_pages_current+0xcc/0x190
>>>> [ 1.389511] [c0000000199df760] [c0000000003c677c]
>>>> pagecache_get_page+0x36c/0x840
>>>> [ 1.389520] [c0000000199df840] [c0000000003c7454]
>>>> grab_cache_page_write_begin+0x34/0x70
>>>> [ 1.389529] [c0000000199df870] [c000000000548504]
>>>> simple_write_begin+0x44/0x100
>>>> [ 1.389537] [c0000000199df8b0] [c0000000003c17fc]
>>>> generic_perform_write+0xec/0x260
>>>> [ 1.389546] [c0000000199df950] [c0000000003c8d70]
>>>> __generic_file_write_iter+0x180/0x2f0
>>>> [ 1.389555] [c0000000199df9b0] [c0000000003c8f8c]
>>>> generic_file_write_iter+0xac/0x110
>>>> [ 1.389563] [c0000000199df9f0] [c0000000004face0]
>>>> do_iter_readv_writev+0x200/0x250
>>>> [ 1.389572] [c0000000199dfa50] [c0000000004fbf78]
>>>> do_iter_write+0xc8/0x270
>>>> [ 1.389579] [c0000000199dfaa0] [c00000000055aa68]
>>>> iter_file_splice_write+0x238/0x500
>>>> [ 1.389587] [c0000000199dfbb0] [c000000000559fac]
>>>> direct_splice_actor+0x4c/0x80
>>>> [ 1.389595] [c0000000199dfbd0] [c0000000005598e0]
>>>> splice_direct_to_actor+0x110/0x320
>>>> [ 1.389603] [c0000000199dfc50] [c000000000559b98]
>>>> do_splice_direct+0xa8/0x110
>>>> [ 1.389611] [c0000000199dfcc0] [c0000000004fe76c]
>>>> vfs_copy_file_range+0x1bc/0x5c0
>>>> [ 1.389619] [c0000000199dfd40] [c0000000004fec5c]
>>>> sys_copy_file_range+0xec/0x270
>>>> [ 1.389626] [c0000000199dfdb0] [c00000000002ddfc]
>>>> system_call_exception+0x10c/0x270
>>>> [ 1.389635] [c0000000199dfe10] [c00000000000d45c]
>>>> system_call_common+0xec/0x278
>>>> [ 1.389643] --- interrupt: c00 at 0x7fffa0ff6bc8
>>>>
>>>>
>>>> Thanks
>>>> Hari
>>>>
>>
>> Hi Kairui
>>
>> > Yes, I did ran into OOM after a panic, when the normal inistramfs is
>> > not built with squash module enbled. And this happens with every
>> > version of the patch, that's why I recommended to enabled squash
>> > module in normal initramfs so fadump can have a much smaller memory
>>
>> With V3, I don't see OOM when the default initramfs is not built with
>> squash module. I guess, the case you are referring to is in comparison
>> to squash enabled vs squash not enabled. I think the additional memory
>> requirement for no squash case is because both initramfs images are
>> unpacked for a while before switch_root can clean up the old
>> initramfs/root. So, that memory usage will go down after switch_root
>> cleans up the oldroot. So, a slight increase in the memory reserved for
>> fadump in the no squash enabled case is likely to resolve the OOM
>> you are referring to.
>
> Hi Hari
>
> The new scratch fadump-init I sent previously, does not support normal
> initramfs without squash module, I've made a new fadump-init that is
> compatible with both squash-enabled and non-squash-enabled normal
> initramfs.
>
>>
>> But with pivot_root approach, no squash default initramfs case is
>> likely to fail even after increasing memory as the same /init script
>> is executing multiple times. I spiked the memory reserved for fadump
>> to 4096MB from 512MB, the log is longer with multiple pivot_root
>> instance errors but it eventually hits OOM even with 4096MB reserved
>> for it (refer attachment)...
>
> With the new init in the attachment, I think it will work just like
> before for non-squash case.
>
>>
>> > usage. I think it should be safe to do so (enabled squash module), so
>> > I think force enabled squash module when fadump is enabled is
>> > acceptable, how do you think?
>>
>> I am fine with squash being used for default initramfs. But I would want
>> the case where squash is not enabled for default initramfs also to be
>> supported though. Behavior above makes me skeptical whether we can
>> support both cases seamlessly with the pivot_root approach proposed.
>>
>> > Also the new scratch init only works when squash module is enabled
>> > since it uses pivot_root, we can make it detect if rootfs is a mount
>> > point and change to use switch_root if we want to be compatible with
>> > normal initramfs built without squash module enabled.
>
>>
>> Right, pivot_root approach has issues when default initramfs is not
>> built with squash.
>>
>> How about replacing "mv" with "cp" in V3 and build default
initramfs
>> without squash by default for now and fix switch_root for squash case
>> (do cleanup and umount overlay) eventually and enable squash by
>> default on default initramfs after that switch_root change?
>
> The reason I tried to use pivot_root for squash case is that, when
> squash module is enabled for normal initramfs, rootfs is not ramfs,
> but a overlay mount of a squashfs. So someone has to do a recursive
> umount of the rootfs, or the content of rootfs is never properly
> cleared. switch_root will just try to `rm` all files in the original
> root, but won't do any umount work... So the loop device used by
> squash module and other resources are unmanaged and leak during the
> whole dump progress, which wastes about 100M (size of the normal
> initramfs).
>
> And if we don't enable squash module for normal initramfs, fadump
> still consumes slightly more memory than kdump (roughly 512M vs 384M).
>
> It turns out not hard to make the fadump-init compatible with both
> squash and non-squash case, I tested the new fadump-init with both
> case and now it seems working very well. I don't have a ppc machine
> available now so I just tested on VM emulating a fadump. I've put the
> patch in the attachment, please have a try and let me know if anything
> is not working.
>
> And I've opened a new pull request to dracut, it's needed if normal
> initramfs is built with squash module enabled:
>
https://github.com/dracutdevs/dracut/pull/1545
>
> And previous PRs are also needed:
>
https://github.com/dracutdevs/dracut/pull/1535
>
https://github.com/dracutdevs/dracut/pull/1532
>
>>
>> But if you insist, I am ok with pivot_root approach as long as default
>> initramfs not having squash enabled case works alright..
>>
>> Thanks
>> Hari
>
> --
> Best Regards,
> Kairui Song
Hi Hari, I found this scratch patch still have some flaw, and here is
a updated version, tested with fadump machines:
Hi Kairui,
Thanks for the scratch patch. It looks much better now.
I have made a few changes (see attachment):
1. Moved is_squash_available() into kdump-lib.sh to reuse.
2. Added a condition in mkfadumprd to build default initramfs with
squash module only if squash is available
3. Changed mv, switch_root utils installations to be conditional.
4. Changed moving files from /fadumproot to use
"$(ls -A /fadumproot)" to avoid running into ".",
".." related
warnings in "mv" command in current form.
- Installing "ls" util would not be an overhead as "ls" is
likely installed already, when squash is not enabled.
5. Added a TODO in mkdumprd to eventually remove the check for
is_fadump_capable() in it.
6. Added your sign off to the patch.
Please take a look. If you are fine with the changes, will post V4.
Thanks
Hari