On Wed, Jun 23, 2021 at 8:17 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
On 23/06/21 1:20 am, Kairui Song wrote:
> On Tue, Jun 22, 2021 at 11:11 PM Kairui Song <kasong(a)redhat.com> wrote:
>>
>> On Mon, Jun 21, 2021 at 6:01 PM Hari Bathini <hbathini(a)linux.ibm.com>
wrote:
>>> On 21/06/21 8:42 am, Kairui Song wrote:
>>>> Hi, Hari
>>>>
>>>> On Fri, Jun 18, 2021 at 11:31 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>> On 17/06/21 8:09 pm, Kairui Song wrote:
>>>>>> On Thu, Jun 17, 2021 at 10:08 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>>> On 16/06/21 1:26 pm, Kairui Song wrote:
>>>>>>>> On Wed, Jun 16, 2021 at 2:32 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>>>>> On 16/06/21 12:42 am, Kairui Song wrote:
>>>>>>>>>> On Tue, Jun 15, 2021 at 2:36 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 14/06/21 3:59 pm, Hari Bathini wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 10/06/21 8:29 pm, Hari Bathini
wrote:
>>>>>>>>>>>>> In case of fadump, the initramfs
image has to be built to boot into
>>>>>>>>>>>>> the production environment as well
as to offload the active crash dump
>>>>>>>>>>>>> to the specified dump target (for
boot after crash). As the same image
>>>>>>>>>>>>> would be used for both boot
scenarios, it could not be built optimally
>>>>>>>>>>>>> while accommodating both cases.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Use --include to include the
initramfs image built for offloading
>>>>>>>>>>>>> active crash dump to the specified
dump target. Also, introduce a new
>>>>>>>>>>>>> out-of-tree dracut module
(99zz-fadumpinit) that installs a customized
>>>>>>>>>>>>> init program while moving the
default /init to /init.dracut. This
>>>>>>>>>>>>> customized init program is leveraged
to isolate fadump image within
>>>>>>>>>>>>> the default initramfs image by
kicking off default boot process
>>>>>>>>>>>>> (exec /init.dracut) for regular boot
scenario and activating fadump
>>>>>>>>>>>>> initramfs image, if the system is
booting after a crash.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Signed-off-by: Hari Bathini
<hbathini(a)linux.ibm.com>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Applying this patch, the initramfs
image built with fadump support
>>>>>>>>>>>>> can be too large, without
>>>>>>>>>>>>>
https://github.com/dracutdevs/dracut/pull/1532
>>>>>>>>>>>>>
>>>>>>>>>>>>> TODO:
>>>>>>>>>>>>> * Fix missing ssh identify file
/root/.ssh/kdump_id_rsa with
>>>>>>>>>>>>> squash not used and remote
(ssh) dump target configured.
>>>>>>>>>>>>> Changes in V3:
>>>>>>>>>>>>> * Removed dependency with squash
module by extracting fadump
>>>>>>>>>>>>> initramfs to /fadumproot
directory.
>>>>>>>>>>>>> * Moved code that builds initramfs
for fadump to mkfadumprd and
>>>>>>>>>>>>> added cleanup support based
on inputs from Kairui.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Changes in V2:
>>>>>>>>>>>>> * Moved kernel modules installation
from install() to installkernel() for
>>>>>>>>>>>>> zz-fadumpinit dracut
module.
>>>>>>>>>>>>> * Updated kexec-tools.spec to
install zz-fadumpinit module files
>>>>>>>>>>>>> appropriately.
>>>>>>>>>>>>> * Fixed error check for capture
initramfs image isolation with dracut.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> dracut-fadump-init-fadump.sh
| 20 ++++++++++++++
>>>>>>>>>>>>> dracut-fadump-module-setup.sh
| 17 ++++++++++++
>>>>>>>>>>>>> dracut-module-setup.sh
| 6 +---
>>>>>>>>>>>>> kdumpctl
| 22 +++-------------
>>>>>>>>>>>>> kexec-tools.spec
| 14 ++++++++++
>>>>>>>>>>>>> mkfadumprd
| 58
>>>>>>>>>>>>>
+++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>>>>> 6 files changed, 115
insertions(+), 22 deletions(-)
>>>>>>>>>>>>> create mode 100644
dracut-fadump-init-fadump.sh
>>>>>>>>>>>>> create mode 100644
dracut-fadump-module-setup.sh
>>>>>>>>>>>>> create mode 100644
mkfadumprd
>>>>>>>>>>>>>
>>>>>>>>>>>>> diff --git
a/dracut-fadump-init-fadump.sh b/dracut-fadump-init-fadump.sh
>>>>>>>>>>>>> new file mode 100644
>>>>>>>>>>>>> index 0000000..736a7dd
>>>>>>>>>>>>> --- /dev/null
>>>>>>>>>>>>> +++ b/dracut-fadump-init-fadump.sh
>>>>>>>>>>>>> @@ -0,0 +1,20 @@
>>>>>>>>>>>>> +#!/bin/sh
>>>>>>>>>>>>> +PATH=/bin:/sbin
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +[ -e /proc/self/mounts ] \
>>>>>>>>>>>>> + || (mkdir -p /proc &&
mount -t proc -o nosuid,noexec,nodev proc
>>>>>>>>>>>>> /proc)
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +grep -q '^sysfs /sys sysfs'
/proc/self/mounts \
>>>>>>>>>>>>> + || (mkdir -p /sys &&
mount -t sysfs -o nosuid,noexec,nodev sysfs
>>>>>>>>>>>>> /sys)
>>>>>>>>>>>>> +
>>>>>>>>>>>>> +# If system is booting after a
crash, switch to initramfs built for
>>>>>>>>>>>>> dump capture
>>>>>>>>>>>>> +if [ -f
/proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
>>>>>>>>>>>>>
/proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>>>>>>>>>>> + mkdir -p /newroot
>>>>>>>>>>>>> + mount -t ramfs ramfs /newroot
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Kairui,
>>>>>>>>>>>
>>>>>>>>>>>>> + mv /fadumproot/* /newroot
>>>>>>>>>>>
>>>>>>>>>>>> Similar to the dracut --include issue,
this would also leave behind
>>>>>>>>>>>> filenames beginning with '.'. As
replacing '*' with '.' won't work,
>>>>>>>>>>>
>>>>>>>>>>> How about the below instead:
>>>>>>>>>>>
>>>>>>>>>>> for FILE in $(ls -A fadumproot); do
mv fadumproot/$FILE newroot; done
>>>>>>>>>>>
>>>>>>>>>>> Need to install 'ls' while setting
up the module though..
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Hari,
>>>>>>>>>
>>>>>>>>> Hi Kairui,
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> After testing with squash module enabled for
both initramfs and the
>>>>>>>>>> embedded fadump initramfs, I think your previous
idea that use `cp`
>>>>>>>>>> consumes same amount of memory as with `mv` is
right.
>>>>>>>>>> Using `mv` will only save memory when the normal
initramfs is not
>>>>>>>>>> using squash module, but I believe we will have
to enable squash
>>>>>>>>>> module for both initramfs to save memory for
fadump, am I right?
>>>>>>>>>
>>>>>>>>> It would be better if we can stick with the same
build parameters for
>>>>>>>>> the default initramfs except for "--add
zz-fadumpinit". So, the way
>>>>>>>>> I look at it, using squash module to build default
initramfs image
>>>>>>>>> must depend on whether it was already with squash or
not. Let me know
>>>>>>>>> your thoughts..
>>>>>>>>
>>>>>>>> I recommended both initramfs to use squash image, this
will help to
>>>>>>>> minimize the memory usage for fadump, so we can use same
crashkernel=
>>>>>>>> value for both fadump and kdump.
>>>>>>>> In my test, when both initramfs uses squash module,
fadump works with
>>>>>>>> 384M of memory, Without squash module enabled for both
initramfs,
>>>>>>>> seems have to reserve 512M of memory. No error or
warning message
>>>>>>>> observed. (just a umount failure, which happens once in
the umount
>>>>>>>> loop in init script, that's expected, we can silence
that later)
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> And base on that assumption (both initramfs uses
squash module), I
>>>>>>>>>> made a scratch `init` file for fadump setup,
this one should work fine
>>>>>>>>>> and consumes less memory, simply call
`switch_root` will left the old
>>>>>>>>>> mount points uncleared so the old squash img is
never dropped and
>>>>>>>>>> consumes ~100m of memory. How do you think?
Other boot init program
>>>>>>>>>> will cleanup old mount points recursively (eg.
systemd), so I think it
>>>>>>>>>> make sense for this script to do something
similar.
>>>>>>>>>> (this require 99squash module to install umount
as well, and
>>>>>>>>>> 99zz-fadump will need to install pivot_root)
>>>>>>>>>
>>>>>>>>> I would like to test this. Other than installing
umount with squash
>>>>>>>>> module, any other changes needed (say in
init-squash.sh)?
>>>>>>>>
>>>>>>>
>>>>>>> Hi Kairui,
>>>>>>>
>>>>>>>> Just install `umount` and include the previous dracut
pull requests we
>>>>>>>> opened, nothing else needed. I'm also testing this
on fadump capable
>>>>>>>> machine, so far everything looks good.
>>>>>>>>
>>>>>>>
>>>>>>> I was trying to test it this afternoon. Ran into OOM issues
with the
>>>>>>> above patch in capture kernel..
>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> #!/bin/sh
>>>>>>>>>> export PATH=/bin:/sbin
>>>>>>>>>>
>>>>>>>>>> [ -e /proc/self/mounts ] \
>>>>>>>>>> || (mkdir -p /proc && mount -t
proc -o nosuid,noexec,nodev proc /proc)
>>>>>>>>>>
>>>>>>>>>> grep -q '^sysfs /sys sysfs'
/proc/self/mounts \
>>>>>>>>>> || (mkdir -p /sys && mount -t
sysfs -o nosuid,noexec,nodev sysfs /sys)
>>>>>>>>>>
>>>>>>>>>> if [ -f /proc/device-tree/rtas/ibm,kernel-dump ]
|| [ -f
>>>>>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ];
then
>>>>>>>>>> mkdir /newroot
>>>>>>>>>> mount -t ramfs ramfs /newroot
>>>>>>>>>>
>>>>>>>>>> cp --reflink=auto --sparse=auto
--preserve=mode,timestamps,links
>>>>>>>>>> -dfr /fadumproot/. /newroot/
>>>>>>>>>>
>>>>>>>>>> mkdir /newroot/sys
>>>>>>>>>> mkdir /newroot/proc
>>>>>>>>>> mkdir /newroot/oldroot
>>>>>>>>>> mount --move /sys /newroot/sys
>>>>>>>>>> mount --move /proc /newroot/proc
>>>>>>>>>>
>>>>>>>>>> cd /newroot || exit 1
>>>>>>>>>> pivot_root . oldroot
>>>>>>>>>>
>>>>>>>>>> # Clean up all mount points in oldroot
>>>>>>>>>> while true; do
>>>>>>>>>> loop=0
>>>>>>>>>> while read -r _ mp _; do
>>>>>>>>>> case $mp in
>>>>>>>>>> /oldroot/* )
>>>>>>>>>> umount "$mp"
&& loop=1
>>>>>>>>>> ;;
>>>>>>>>>> esac
>>>>>>>>>> done < /proc/mounts
>>>>>>>>>> [ $loop = 1 ] || break
>>>>>>>>>> done
>>>>>>>>>> umount -l oldroot
>>>>>>>>>>
>>>>>>>
>>>>>>>>>> # Jump to initramfs built for dump
capture
>>>>>>>>>> exec /init
>>>>>>>
>>>>>>> Must have to do with the above. At the least, "exec
/newroot/init".
>>>>>>> Also, isn't this missing a chroot..
>>>>>>
>>>>>> Hi Hari,
>>>>>>
>>>>>> Did you miss any pull request from dracut? Like:
>>>>>>
https://github.com/dracutdevs/dracut/pull/1532
>>>>> >
>>>>> Sorry, I could have been clearer. When squash module is used on
both
>>>>> default and fadump initramfs, it is working but when the default
>>>>> initramfs is built without squash enabled, running into the OOM
>>>>> issues in capture kernel. Did you get a chance to try this
scenario?
>>>>> If yes, no issues in this scenario as well? fwiw, I see the below
>>>>> trace in capture kernel when default initramfs is not built with
>>>>> squash module but only "--add zz-fadumpinit":
>>>>>
>>>>> [ 1.191970] Freeing unused kernel memory: 6528K
>>>>> [ 1.191978] This architecture does not have kernel
memory protection.
>>>>> [ 1.191988] Run /init as init process
>>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>>> /init: line 44: /proc/mounts: No such file or directory
>>>>> umount: oldroot: umount failed: No such file or
directory.
>>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>>> /init: line 44: /proc/mounts: No such file or directory
>>>>> umount: oldroot: umount failed: No such file or
directory.
>>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>>> /init: line 44: /proc/mounts: No such file or directory
>>>>> umount: oldroot: umount failed: No such file or
directory.
>>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>>> /init: line 44: /proc/mounts: No such file or directory
>>>>> umount: oldroot: umount failed: No such file or
directory.
>>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>>> /init: line 44: /proc/mounts: No such file or directory
>>>>> umount: oldroot: umount failed: No such file or
directory.
>>>>> pivot_root: failed to change root from `.' to
`oldroot': Invalid argument
>>>>> /init: line 44: /proc/mounts: No such file or directory
>>>>> umount: oldroot: umount failed: No such file or
directory.
>>>>> [ 1.389399] cp invoked oom-killer:
gfp_mask=0x100cc2(GFP_HIGHUSER),
>>>>> order=0, oom_score_adj=0
>>>>> [ 1.389435] CPU: 6 PID: 424 Comm: cp Not tainted
>>>>> 5.12.10-300.fc34.ppc64le #1
>>>>> [ 1.389443] Call Trace:
>>>>> [ 1.389446] [c0000000199df310] [c0000000009e881c]
>>>>> dump_stack+0xc0/0x104 (unreliable)
>>>>> [ 1.389460] [c0000000199df360] [c0000000003d12a8]
dump_header+0x64/0x284
>>>>> [ 1.389469] [c0000000199df3f0] [c0000000003cf970]
>>>>> oom_kill_process+0x2c0/0x2d0
>>>>> [ 1.389477] [c0000000199df430] [c0000000003d0c14]
>>>>> out_of_memory+0x224/0x780
>>>>> [ 1.389485] [c0000000199df4d0] [c00000000045c7f0]
>>>>> __alloc_pages_slowpath.constprop.0+0xc60/0xda0
>>>>> [ 1.389494] [c0000000199df680] [c00000000045cc50]
>>>>> __alloc_pages_nodemask+0x320/0x3d0
>>>>> [ 1.389503] [c0000000199df710] [c00000000048b95c]
>>>>> alloc_pages_current+0xcc/0x190
>>>>> [ 1.389511] [c0000000199df760] [c0000000003c677c]
>>>>> pagecache_get_page+0x36c/0x840
>>>>> [ 1.389520] [c0000000199df840] [c0000000003c7454]
>>>>> grab_cache_page_write_begin+0x34/0x70
>>>>> [ 1.389529] [c0000000199df870] [c000000000548504]
>>>>> simple_write_begin+0x44/0x100
>>>>> [ 1.389537] [c0000000199df8b0] [c0000000003c17fc]
>>>>> generic_perform_write+0xec/0x260
>>>>> [ 1.389546] [c0000000199df950] [c0000000003c8d70]
>>>>> __generic_file_write_iter+0x180/0x2f0
>>>>> [ 1.389555] [c0000000199df9b0] [c0000000003c8f8c]
>>>>> generic_file_write_iter+0xac/0x110
>>>>> [ 1.389563] [c0000000199df9f0] [c0000000004face0]
>>>>> do_iter_readv_writev+0x200/0x250
>>>>> [ 1.389572] [c0000000199dfa50] [c0000000004fbf78]
>>>>> do_iter_write+0xc8/0x270
>>>>> [ 1.389579] [c0000000199dfaa0] [c00000000055aa68]
>>>>> iter_file_splice_write+0x238/0x500
>>>>> [ 1.389587] [c0000000199dfbb0] [c000000000559fac]
>>>>> direct_splice_actor+0x4c/0x80
>>>>> [ 1.389595] [c0000000199dfbd0] [c0000000005598e0]
>>>>> splice_direct_to_actor+0x110/0x320
>>>>> [ 1.389603] [c0000000199dfc50] [c000000000559b98]
>>>>> do_splice_direct+0xa8/0x110
>>>>> [ 1.389611] [c0000000199dfcc0] [c0000000004fe76c]
>>>>> vfs_copy_file_range+0x1bc/0x5c0
>>>>> [ 1.389619] [c0000000199dfd40] [c0000000004fec5c]
>>>>> sys_copy_file_range+0xec/0x270
>>>>> [ 1.389626] [c0000000199dfdb0] [c00000000002ddfc]
>>>>> system_call_exception+0x10c/0x270
>>>>> [ 1.389635] [c0000000199dfe10] [c00000000000d45c]
>>>>> system_call_common+0xec/0x278
>>>>> [ 1.389643] --- interrupt: c00 at 0x7fffa0ff6bc8
>>>>>
>>>>>
>>>>> Thanks
>>>>> Hari
>>>>>
>>>
>>> Hi Kairui
>>>
>>> > Yes, I did ran into OOM after a panic, when the normal inistramfs is
>>> > not built with squash module enbled. And this happens with every
>>> > version of the patch, that's why I recommended to enabled squash
>>> > module in normal initramfs so fadump can have a much smaller memory
>>>
>>> With V3, I don't see OOM when the default initramfs is not built with
>>> squash module. I guess, the case you are referring to is in comparison
>>> to squash enabled vs squash not enabled. I think the additional memory
>>> requirement for no squash case is because both initramfs images are
>>> unpacked for a while before switch_root can clean up the old
>>> initramfs/root. So, that memory usage will go down after switch_root
>>> cleans up the oldroot. So, a slight increase in the memory reserved for
>>> fadump in the no squash enabled case is likely to resolve the OOM
>>> you are referring to.
>>
>> Hi Hari
>>
>> The new scratch fadump-init I sent previously, does not support normal
>> initramfs without squash module, I've made a new fadump-init that is
>> compatible with both squash-enabled and non-squash-enabled normal
>> initramfs.
>>
>>>
>>> But with pivot_root approach, no squash default initramfs case is
>>> likely to fail even after increasing memory as the same /init script
>>> is executing multiple times. I spiked the memory reserved for fadump
>>> to 4096MB from 512MB, the log is longer with multiple pivot_root
>>> instance errors but it eventually hits OOM even with 4096MB reserved
>>> for it (refer attachment)...
>>
>> With the new init in the attachment, I think it will work just like
>> before for non-squash case.
>>
>>>
>>> > usage. I think it should be safe to do so (enabled squash module),
so
>>> > I think force enabled squash module when fadump is enabled is
>>> > acceptable, how do you think?
>>>
>>> I am fine with squash being used for default initramfs. But I would want
>>> the case where squash is not enabled for default initramfs also to be
>>> supported though. Behavior above makes me skeptical whether we can
>>> support both cases seamlessly with the pivot_root approach proposed.
>>>
>>> > Also the new scratch init only works when squash module is enabled
>>> > since it uses pivot_root, we can make it detect if rootfs is a mount
>>> > point and change to use switch_root if we want to be compatible with
>>> > normal initramfs built without squash module enabled.
>>
>>>
>>> Right, pivot_root approach has issues when default initramfs is not
>>> built with squash.
>>>
>>> How about replacing "mv" with "cp" in V3 and build
default initramfs
>>> without squash by default for now and fix switch_root for squash case
>>> (do cleanup and umount overlay) eventually and enable squash by
>>> default on default initramfs after that switch_root change?
>>
>> The reason I tried to use pivot_root for squash case is that, when
>> squash module is enabled for normal initramfs, rootfs is not ramfs,
>> but a overlay mount of a squashfs. So someone has to do a recursive
>> umount of the rootfs, or the content of rootfs is never properly
>> cleared. switch_root will just try to `rm` all files in the original
>> root, but won't do any umount work... So the loop device used by
>> squash module and other resources are unmanaged and leak during the
>> whole dump progress, which wastes about 100M (size of the normal
>> initramfs).
>>
>> And if we don't enable squash module for normal initramfs, fadump
>> still consumes slightly more memory than kdump (roughly 512M vs 384M).
>>
>> It turns out not hard to make the fadump-init compatible with both
>> squash and non-squash case, I tested the new fadump-init with both
>> case and now it seems working very well. I don't have a ppc machine
>> available now so I just tested on VM emulating a fadump. I've put the
>> patch in the attachment, please have a try and let me know if anything
>> is not working.
>>
>> And I've opened a new pull request to dracut, it's needed if normal
>> initramfs is built with squash module enabled:
>>
https://github.com/dracutdevs/dracut/pull/1545
>>
>> And previous PRs are also needed:
>>
https://github.com/dracutdevs/dracut/pull/1535
>>
https://github.com/dracutdevs/dracut/pull/1532
>>
>>>
>>> But if you insist, I am ok with pivot_root approach as long as default
>>> initramfs not having squash enabled case works alright..
>>>
>>> Thanks
>>> Hari
>>
>> --
>> Best Regards,
>> Kairui Song
>
> Hi Hari, I found this scratch patch still have some flaw, and here is
> a updated version, tested with fadump machines:
Hi Kairui,
Thanks for the scratch patch. It looks much better now.
I have made a few changes (see attachment):
1. Moved is_squash_available() into kdump-lib.sh to reuse.
2. Added a condition in mkfadumprd to build default initramfs with
squash module only if squash is available
3. Changed mv, switch_root utils installations to be conditional.
4. Changed moving files from /fadumproot to use
"$(ls -A /fadumproot)" to avoid running into ".",
".." related
warnings in "mv" command in current form.
- Installing "ls" util would not be an overhead as "ls" is
likely installed already, when squash is not enabled.
5. Added a TODO in mkdumprd to eventually remove the check for
is_fadump_capable() in it.
6. Added your sign off to the patch.
Please take a look. If you are fine with the changes, will post V4.
Thanks
Hari