On Mon, Jun 21, 2021 at 6:01 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
On 21/06/21 8:42 am, Kairui Song wrote:
> Hi, Hari
>
> On Fri, Jun 18, 2021 at 11:31 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
>> On 17/06/21 8:09 pm, Kairui Song wrote:
>>> On Thu, Jun 17, 2021 at 10:08 PM Hari Bathini <hbathini(a)linux.ibm.com>
wrote:
>>>> On 16/06/21 1:26 pm, Kairui Song wrote:
>>>>> On Wed, Jun 16, 2021 at 2:32 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>> On 16/06/21 12:42 am, Kairui Song wrote:
>>>>>>> On Tue, Jun 15, 2021 at 2:36 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 14/06/21 3:59 pm, Hari Bathini wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 10/06/21 8:29 pm, Hari Bathini wrote:
>>>>>>>>>> In case of fadump, the initramfs image has to be
built to boot into
>>>>>>>>>> the production environment as well as to offload
the active crash dump
>>>>>>>>>> to the specified dump target (for boot after
crash). As the same image
>>>>>>>>>> would be used for both boot scenarios, it could
not be built optimally
>>>>>>>>>> while accommodating both cases.
>>>>>>>>>>
>>>>>>>>>> Use --include to include the initramfs image
built for offloading
>>>>>>>>>> active crash dump to the specified dump target.
Also, introduce a new
>>>>>>>>>> out-of-tree dracut module (99zz-fadumpinit) that
installs a customized
>>>>>>>>>> init program while moving the default /init to
/init.dracut. This
>>>>>>>>>> customized init program is leveraged to isolate
fadump image within
>>>>>>>>>> the default initramfs image by kicking off
default boot process
>>>>>>>>>> (exec /init.dracut) for regular boot scenario
and activating fadump
>>>>>>>>>> initramfs image, if the system is booting after
a crash.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Hari Bathini
<hbathini(a)linux.ibm.com>
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> * Applying this patch, the initramfs image built
with fadump support
>>>>>>>>>> can be too large, without
>>>>>>>>>>
https://github.com/dracutdevs/dracut/pull/1532
>>>>>>>>>>
>>>>>>>>>> TODO:
>>>>>>>>>> * Fix missing ssh identify file
/root/.ssh/kdump_id_rsa with
>>>>>>>>>> squash not used and remote (ssh) dump
target configured.
>>>>>>>>>> Changes in V3:
>>>>>>>>>> * Removed dependency with squash module by
extracting fadump
>>>>>>>>>> initramfs to /fadumproot directory.
>>>>>>>>>> * Moved code that builds initramfs for fadump to
mkfadumprd and
>>>>>>>>>> added cleanup support based on inputs
from Kairui.
>>>>>>>>>>
>>>>>>>>>> Changes in V2:
>>>>>>>>>> * Moved kernel modules installation from
install() to installkernel() for
>>>>>>>>>> zz-fadumpinit dracut module.
>>>>>>>>>> * Updated kexec-tools.spec to install
zz-fadumpinit module files
>>>>>>>>>> appropriately.
>>>>>>>>>> * Fixed error check for capture initramfs image
isolation with dracut.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> dracut-fadump-init-fadump.sh | 20
++++++++++++++
>>>>>>>>>> dracut-fadump-module-setup.sh | 17
++++++++++++
>>>>>>>>>> dracut-module-setup.sh | 6 +---
>>>>>>>>>> kdumpctl | 22
+++-------------
>>>>>>>>>> kexec-tools.spec | 14
++++++++++
>>>>>>>>>> mkfadumprd | 58
>>>>>>>>>> +++++++++++++++++++++++++++++++++++++++++
>>>>>>>>>> 6 files changed, 115 insertions(+), 22
deletions(-)
>>>>>>>>>> create mode 100644
dracut-fadump-init-fadump.sh
>>>>>>>>>> create mode 100644
dracut-fadump-module-setup.sh
>>>>>>>>>> create mode 100644 mkfadumprd
>>>>>>>>>>
>>>>>>>>>> diff --git a/dracut-fadump-init-fadump.sh
b/dracut-fadump-init-fadump.sh
>>>>>>>>>> new file mode 100644
>>>>>>>>>> index 0000000..736a7dd
>>>>>>>>>> --- /dev/null
>>>>>>>>>> +++ b/dracut-fadump-init-fadump.sh
>>>>>>>>>> @@ -0,0 +1,20 @@
>>>>>>>>>> +#!/bin/sh
>>>>>>>>>> +PATH=/bin:/sbin
>>>>>>>>>> +
>>>>>>>>>> +[ -e /proc/self/mounts ] \
>>>>>>>>>> + || (mkdir -p /proc && mount -t proc
-o nosuid,noexec,nodev proc
>>>>>>>>>> /proc)
>>>>>>>>>> +
>>>>>>>>>> +grep -q '^sysfs /sys sysfs'
/proc/self/mounts \
>>>>>>>>>> + || (mkdir -p /sys && mount -t sysfs
-o nosuid,noexec,nodev sysfs
>>>>>>>>>> /sys)
>>>>>>>>>> +
>>>>>>>>>> +# If system is booting after a crash, switch to
initramfs built for
>>>>>>>>>> dump capture
>>>>>>>>>> +if [ -f /proc/device-tree/rtas/ibm,kernel-dump
] || [ -f
>>>>>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ];
then
>>>>>>>>>> + mkdir -p /newroot
>>>>>>>>>> + mount -t ramfs ramfs /newroot
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Kairui,
>>>>>>>>
>>>>>>>>>> + mv /fadumproot/* /newroot
>>>>>>>>
>>>>>>>>> Similar to the dracut --include issue, this would
also leave behind
>>>>>>>>> filenames beginning with '.'. As replacing
'*' with '.' won't work,
>>>>>>>>
>>>>>>>> How about the below instead:
>>>>>>>>
>>>>>>>> for FILE in $(ls -A fadumproot); do mv
fadumproot/$FILE newroot; done
>>>>>>>>
>>>>>>>> Need to install 'ls' while setting up the module
though..
>>>>>>>>
>>>>>>>
>>>>>>> Hi Hari,
>>>>>>
>>>>>> Hi Kairui,
>>>>>>
>>>>>>>
>>>>>>> After testing with squash module enabled for both initramfs
and the
>>>>>>> embedded fadump initramfs, I think your previous idea that
use `cp`
>>>>>>> consumes same amount of memory as with `mv` is right.
>>>>>>> Using `mv` will only save memory when the normal initramfs
is not
>>>>>>> using squash module, but I believe we will have to enable
squash
>>>>>>> module for both initramfs to save memory for fadump, am I
right?
>>>>>>
>>>>>> It would be better if we can stick with the same build
parameters for
>>>>>> the default initramfs except for "--add
zz-fadumpinit". So, the way
>>>>>> I look at it, using squash module to build default initramfs
image
>>>>>> must depend on whether it was already with squash or not. Let me
know
>>>>>> your thoughts..
>>>>>
>>>>> I recommended both initramfs to use squash image, this will help to
>>>>> minimize the memory usage for fadump, so we can use same
crashkernel=
>>>>> value for both fadump and kdump.
>>>>> In my test, when both initramfs uses squash module, fadump works
with
>>>>> 384M of memory, Without squash module enabled for both initramfs,
>>>>> seems have to reserve 512M of memory. No error or warning message
>>>>> observed. (just a umount failure, which happens once in the umount
>>>>> loop in init script, that's expected, we can silence that
later)
>>>>>
>>>>>>
>>>>>>>
>>>>>>> And base on that assumption (both initramfs uses squash
module), I
>>>>>>> made a scratch `init` file for fadump setup, this one should
work fine
>>>>>>> and consumes less memory, simply call `switch_root` will
left the old
>>>>>>> mount points uncleared so the old squash img is never
dropped and
>>>>>>> consumes ~100m of memory. How do you think? Other boot init
program
>>>>>>> will cleanup old mount points recursively (eg. systemd), so
I think it
>>>>>>> make sense for this script to do something similar.
>>>>>>> (this require 99squash module to install umount as well,
and
>>>>>>> 99zz-fadump will need to install pivot_root)
>>>>>>
>>>>>> I would like to test this. Other than installing umount with
squash
>>>>>> module, any other changes needed (say in init-squash.sh)?
>>>>>
>>>>
>>>> Hi Kairui,
>>>>
>>>>> Just install `umount` and include the previous dracut pull requests
we
>>>>> opened, nothing else needed. I'm also testing this on fadump
capable
>>>>> machine, so far everything looks good.
>>>>>
>>>>
>>>> I was trying to test it this afternoon. Ran into OOM issues with the
>>>> above patch in capture kernel..
>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> #!/bin/sh
>>>>>>> export PATH=/bin:/sbin
>>>>>>>
>>>>>>> [ -e /proc/self/mounts ] \
>>>>>>> || (mkdir -p /proc && mount -t proc -o
nosuid,noexec,nodev proc /proc)
>>>>>>>
>>>>>>> grep -q '^sysfs /sys sysfs' /proc/self/mounts \
>>>>>>> || (mkdir -p /sys && mount -t sysfs -o
nosuid,noexec,nodev sysfs /sys)
>>>>>>>
>>>>>>> if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
>>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>>>>> mkdir /newroot
>>>>>>> mount -t ramfs ramfs /newroot
>>>>>>>
>>>>>>> cp --reflink=auto --sparse=auto
--preserve=mode,timestamps,links
>>>>>>> -dfr /fadumproot/. /newroot/
>>>>>>>
>>>>>>> mkdir /newroot/sys
>>>>>>> mkdir /newroot/proc
>>>>>>> mkdir /newroot/oldroot
>>>>>>> mount --move /sys /newroot/sys
>>>>>>> mount --move /proc /newroot/proc
>>>>>>>
>>>>>>> cd /newroot || exit 1
>>>>>>> pivot_root . oldroot
>>>>>>>
>>>>>>> # Clean up all mount points in oldroot
>>>>>>> while true; do
>>>>>>> loop=0
>>>>>>> while read -r _ mp _; do
>>>>>>> case $mp in
>>>>>>> /oldroot/* )
>>>>>>> umount "$mp" &&
loop=1
>>>>>>> ;;
>>>>>>> esac
>>>>>>> done < /proc/mounts
>>>>>>> [ $loop = 1 ] || break
>>>>>>> done
>>>>>>> umount -l oldroot
>>>>>>>
>>>>
>>>>>>> # Jump to initramfs built for dump capture
>>>>>>> exec /init
>>>>
>>>> Must have to do with the above. At the least, "exec
/newroot/init".
>>>> Also, isn't this missing a chroot..
>>>
>>> Hi Hari,
>>>
>>> Did you miss any pull request from dracut? Like:
>>>
https://github.com/dracutdevs/dracut/pull/1532
>> >
>> Sorry, I could have been clearer. When squash module is used on both
>> default and fadump initramfs, it is working but when the default
>> initramfs is built without squash enabled, running into the OOM
>> issues in capture kernel. Did you get a chance to try this scenario?
>> If yes, no issues in this scenario as well? fwiw, I see the below
>> trace in capture kernel when default initramfs is not built with
>> squash module but only "--add zz-fadumpinit":
>>
>> [ 1.191970] Freeing unused kernel memory: 6528K
>> [ 1.191978] This architecture does not have kernel memory
protection.
>> [ 1.191988] Run /init as init process
>> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
>> /init: line 44: /proc/mounts: No such file or directory
>> umount: oldroot: umount failed: No such file or directory.
>> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
>> /init: line 44: /proc/mounts: No such file or directory
>> umount: oldroot: umount failed: No such file or directory.
>> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
>> /init: line 44: /proc/mounts: No such file or directory
>> umount: oldroot: umount failed: No such file or directory.
>> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
>> /init: line 44: /proc/mounts: No such file or directory
>> umount: oldroot: umount failed: No such file or directory.
>> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
>> /init: line 44: /proc/mounts: No such file or directory
>> umount: oldroot: umount failed: No such file or directory.
>> pivot_root: failed to change root from `.' to `oldroot':
Invalid argument
>> /init: line 44: /proc/mounts: No such file or directory
>> umount: oldroot: umount failed: No such file or directory.
>> [ 1.389399] cp invoked oom-killer: gfp_mask=0x100cc2(GFP_HIGHUSER),
>> order=0, oom_score_adj=0
>> [ 1.389435] CPU: 6 PID: 424 Comm: cp Not tainted
>> 5.12.10-300.fc34.ppc64le #1
>> [ 1.389443] Call Trace:
>> [ 1.389446] [c0000000199df310] [c0000000009e881c]
>> dump_stack+0xc0/0x104 (unreliable)
>> [ 1.389460] [c0000000199df360] [c0000000003d12a8]
dump_header+0x64/0x284
>> [ 1.389469] [c0000000199df3f0] [c0000000003cf970]
>> oom_kill_process+0x2c0/0x2d0
>> [ 1.389477] [c0000000199df430] [c0000000003d0c14]
>> out_of_memory+0x224/0x780
>> [ 1.389485] [c0000000199df4d0] [c00000000045c7f0]
>> __alloc_pages_slowpath.constprop.0+0xc60/0xda0
>> [ 1.389494] [c0000000199df680] [c00000000045cc50]
>> __alloc_pages_nodemask+0x320/0x3d0
>> [ 1.389503] [c0000000199df710] [c00000000048b95c]
>> alloc_pages_current+0xcc/0x190
>> [ 1.389511] [c0000000199df760] [c0000000003c677c]
>> pagecache_get_page+0x36c/0x840
>> [ 1.389520] [c0000000199df840] [c0000000003c7454]
>> grab_cache_page_write_begin+0x34/0x70
>> [ 1.389529] [c0000000199df870] [c000000000548504]
>> simple_write_begin+0x44/0x100
>> [ 1.389537] [c0000000199df8b0] [c0000000003c17fc]
>> generic_perform_write+0xec/0x260
>> [ 1.389546] [c0000000199df950] [c0000000003c8d70]
>> __generic_file_write_iter+0x180/0x2f0
>> [ 1.389555] [c0000000199df9b0] [c0000000003c8f8c]
>> generic_file_write_iter+0xac/0x110
>> [ 1.389563] [c0000000199df9f0] [c0000000004face0]
>> do_iter_readv_writev+0x200/0x250
>> [ 1.389572] [c0000000199dfa50] [c0000000004fbf78]
>> do_iter_write+0xc8/0x270
>> [ 1.389579] [c0000000199dfaa0] [c00000000055aa68]
>> iter_file_splice_write+0x238/0x500
>> [ 1.389587] [c0000000199dfbb0] [c000000000559fac]
>> direct_splice_actor+0x4c/0x80
>> [ 1.389595] [c0000000199dfbd0] [c0000000005598e0]
>> splice_direct_to_actor+0x110/0x320
>> [ 1.389603] [c0000000199dfc50] [c000000000559b98]
>> do_splice_direct+0xa8/0x110
>> [ 1.389611] [c0000000199dfcc0] [c0000000004fe76c]
>> vfs_copy_file_range+0x1bc/0x5c0
>> [ 1.389619] [c0000000199dfd40] [c0000000004fec5c]
>> sys_copy_file_range+0xec/0x270
>> [ 1.389626] [c0000000199dfdb0] [c00000000002ddfc]
>> system_call_exception+0x10c/0x270
>> [ 1.389635] [c0000000199dfe10] [c00000000000d45c]
>> system_call_common+0xec/0x278
>> [ 1.389643] --- interrupt: c00 at 0x7fffa0ff6bc8
>>
>>
>> Thanks
>> Hari
>>
Hi Kairui
> Yes, I did ran into OOM after a panic, when the normal inistramfs is
> not built with squash module enbled. And this happens with every
> version of the patch, that's why I recommended to enabled squash
> module in normal initramfs so fadump can have a much smaller memory
With V3, I don't see OOM when the default initramfs is not built with
squash module. I guess, the case you are referring to is in comparison
to squash enabled vs squash not enabled. I think the additional memory
requirement for no squash case is because both initramfs images are
unpacked for a while before switch_root can clean up the old
initramfs/root. So, that memory usage will go down after switch_root
cleans up the oldroot. So, a slight increase in the memory reserved for
fadump in the no squash enabled case is likely to resolve the OOM
you are referring to.
Hi Hari
The new scratch fadump-init I sent previously, does not support normal
initramfs without squash module, I've made a new fadump-init that is
compatible with both squash-enabled and non-squash-enabled normal
initramfs.
But with pivot_root approach, no squash default initramfs case is
likely to fail even after increasing memory as the same /init script
is executing multiple times. I spiked the memory reserved for fadump
to 4096MB from 512MB, the log is longer with multiple pivot_root
instance errors but it eventually hits OOM even with 4096MB reserved
for it (refer attachment)...
With the new init in the attachment, I think it will work just like
before for non-squash case.
> usage. I think it should be safe to do so (enabled squash module), so
> I think force enabled squash module when fadump is enabled is
> acceptable, how do you think?
I am fine with squash being used for default initramfs. But I would want
the case where squash is not enabled for default initramfs also to be
supported though. Behavior above makes me skeptical whether we can
support both cases seamlessly with the pivot_root approach proposed.
> Also the new scratch init only works when squash module is enabled
> since it uses pivot_root, we can make it detect if rootfs is a mount
> point and change to use switch_root if we want to be compatible with
> normal initramfs built without squash module enabled.
Right, pivot_root approach has issues when default initramfs is not
built with squash.
How about replacing "mv" with "cp" in V3 and build default initramfs
without squash by default for now and fix switch_root for squash case
(do cleanup and umount overlay) eventually and enable squash by
default on default initramfs after that switch_root change?
The reason I tried to use pivot_root for squash case is that, when
squash module is enabled for normal initramfs, rootfs is not ramfs,
but a overlay mount of a squashfs. So someone has to do a recursive
umount of the rootfs, or the content of rootfs is never properly
cleared. switch_root will just try to `rm` all files in the original
root, but won't do any umount work... So the loop device used by
squash module and other resources are unmanaged and leak during the
whole dump progress, which wastes about 100M (size of the normal
initramfs).
And if we don't enable squash module for normal initramfs, fadump
still consumes slightly more memory than kdump (roughly 512M vs 384M).
It turns out not hard to make the fadump-init compatible with both
squash and non-squash case, I tested the new fadump-init with both
case and now it seems working very well. I don't have a ppc machine
available now so I just tested on VM emulating a fadump. I've put the
patch in the attachment, please have a try and let me know if anything
is not working.
And I've opened a new pull request to dracut, it's needed if normal
initramfs is built with squash module enabled:
https://github.com/dracutdevs/dracut/pull/1545
And previous PRs are also needed:
https://github.com/dracutdevs/dracut/pull/1535
https://github.com/dracutdevs/dracut/pull/1532
But if you insist, I am ok with pivot_root approach as long as default
initramfs not having squash enabled case works alright..
Thanks
Hari
--
Best Regards,
Kairui Song