On Thu, Jun 17, 2021 at 10:08 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
> On 16/06/21 1:26 pm, Kairui Song wrote:
>> On Wed, Jun 16, 2021 at 2:32 PM Hari Bathini <hbathini(a)linux.ibm.com>
wrote:
>>> On 16/06/21 12:42 am, Kairui Song wrote:
>>>> On Tue, Jun 15, 2021 at 2:36 PM Hari Bathini
<hbathini(a)linux.ibm.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 14/06/21 3:59 pm, Hari Bathini wrote:
>>>>>>
>>>>>>
>>>>>> On 10/06/21 8:29 pm, Hari Bathini wrote:
>>>>>>> In case of fadump, the initramfs image has to be built to
boot into
>>>>>>> the production environment as well as to offload the active
crash dump
>>>>>>> to the specified dump target (for boot after crash). As the
same image
>>>>>>> would be used for both boot scenarios, it could not be built
optimally
>>>>>>> while accommodating both cases.
>>>>>>>
>>>>>>> Use --include to include the initramfs image built for
offloading
>>>>>>> active crash dump to the specified dump target. Also,
introduce a new
>>>>>>> out-of-tree dracut module (99zz-fadumpinit) that installs a
customized
>>>>>>> init program while moving the default /init to /init.dracut.
This
>>>>>>> customized init program is leveraged to isolate fadump image
within
>>>>>>> the default initramfs image by kicking off default boot
process
>>>>>>> (exec /init.dracut) for regular boot scenario and activating
fadump
>>>>>>> initramfs image, if the system is booting after a crash.
>>>>>>>
>>>>>>> Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
>>>>>>> ---
>>>>>>>
>>>>>>> * Applying this patch, the initramfs image built with fadump
support
>>>>>>> can be too large, without
>>>>>>>
https://github.com/dracutdevs/dracut/pull/1532
>>>>>>>
>>>>>>> TODO:
>>>>>>> * Fix missing ssh identify file /root/.ssh/kdump_id_rsa with
>>>>>>> squash not used and remote (ssh) dump target
configured.
>>>>>>> Changes in V3:
>>>>>>> * Removed dependency with squash module by extracting fadump
>>>>>>> initramfs to /fadumproot directory.
>>>>>>> * Moved code that builds initramfs for fadump to mkfadumprd
and
>>>>>>> added cleanup support based on inputs from Kairui.
>>>>>>>
>>>>>>> Changes in V2:
>>>>>>> * Moved kernel modules installation from install() to
installkernel() for
>>>>>>> zz-fadumpinit dracut module.
>>>>>>> * Updated kexec-tools.spec to install zz-fadumpinit module
files
>>>>>>> appropriately.
>>>>>>> * Fixed error check for capture initramfs image isolation
with dracut.
>>>>>>>
>>>>>>>
>>>>>>> dracut-fadump-init-fadump.sh | 20 ++++++++++++++
>>>>>>> dracut-fadump-module-setup.sh | 17 ++++++++++++
>>>>>>> dracut-module-setup.sh | 6 +---
>>>>>>> kdumpctl | 22 +++-------------
>>>>>>> kexec-tools.spec | 14 ++++++++++
>>>>>>> mkfadumprd | 58
>>>>>>> +++++++++++++++++++++++++++++++++++++++++
>>>>>>> 6 files changed, 115 insertions(+), 22 deletions(-)
>>>>>>> create mode 100644 dracut-fadump-init-fadump.sh
>>>>>>> create mode 100644 dracut-fadump-module-setup.sh
>>>>>>> create mode 100644 mkfadumprd
>>>>>>>
>>>>>>> diff --git a/dracut-fadump-init-fadump.sh
b/dracut-fadump-init-fadump.sh
>>>>>>> new file mode 100644
>>>>>>> index 0000000..736a7dd
>>>>>>> --- /dev/null
>>>>>>> +++ b/dracut-fadump-init-fadump.sh
>>>>>>> @@ -0,0 +1,20 @@
>>>>>>> +#!/bin/sh
>>>>>>> +PATH=/bin:/sbin
>>>>>>> +
>>>>>>> +[ -e /proc/self/mounts ] \
>>>>>>> + || (mkdir -p /proc && mount -t proc -o
nosuid,noexec,nodev proc
>>>>>>> /proc)
>>>>>>> +
>>>>>>> +grep -q '^sysfs /sys sysfs' /proc/self/mounts \
>>>>>>> + || (mkdir -p /sys && mount -t sysfs -o
nosuid,noexec,nodev sysfs
>>>>>>> /sys)
>>>>>>> +
>>>>>>> +# If system is booting after a crash, switch to initramfs
built for
>>>>>>> dump capture
>>>>>>> +if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
>>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>>>>> + mkdir -p /newroot
>>>>>>> + mount -t ramfs ramfs /newroot
>>>>>
>>>>>
>>>>> Hi Kairui,
>>>>>
>>>>>>> + mv /fadumproot/* /newroot
>>>>>
>>>>>> Similar to the dracut --include issue, this would also leave
behind
>>>>>> filenames beginning with '.'. As replacing '*'
with '.' won't work,
>>>>>
>>>>> How about the below instead:
>>>>>
>>>>> for FILE in $(ls -A fadumproot); do mv fadumproot/$FILE
newroot; done
>>>>>
>>>>> Need to install 'ls' while setting up the module though..
>>>>>
>>>>
>>>> Hi Hari,
>>>
>>> Hi Kairui,
>>>
>>>>
>>>> After testing with squash module enabled for both initramfs and the
>>>> embedded fadump initramfs, I think your previous idea that use `cp`
>>>> consumes same amount of memory as with `mv` is right.
>>>> Using `mv` will only save memory when the normal initramfs is not
>>>> using squash module, but I believe we will have to enable squash
>>>> module for both initramfs to save memory for fadump, am I right?
>>>
>>> It would be better if we can stick with the same build parameters for
>>> the default initramfs except for "--add zz-fadumpinit". So, the
way
>>> I look at it, using squash module to build default initramfs image
>>> must depend on whether it was already with squash or not. Let me know
>>> your thoughts..
>>
>> I recommended both initramfs to use squash image, this will help to
>> minimize the memory usage for fadump, so we can use same crashkernel=
>> value for both fadump and kdump.
>> In my test, when both initramfs uses squash module, fadump works with
>> 384M of memory, Without squash module enabled for both initramfs,
>> seems have to reserve 512M of memory. No error or warning message
>> observed. (just a umount failure, which happens once in the umount
>> loop in init script, that's expected, we can silence that later)
>>
>>>
>>>>
>>>> And base on that assumption (both initramfs uses squash module), I
>>>> made a scratch `init` file for fadump setup, this one should work fine
>>>> and consumes less memory, simply call `switch_root` will left the old
>>>> mount points uncleared so the old squash img is never dropped and
>>>> consumes ~100m of memory. How do you think? Other boot init program
>>>> will cleanup old mount points recursively (eg. systemd), so I think it
>>>> make sense for this script to do something similar.
>>>> (this require 99squash module to install umount as well, and
>>>> 99zz-fadump will need to install pivot_root)
>>>
>>> I would like to test this. Other than installing umount with squash
>>> module, any other changes needed (say in init-squash.sh)?
>>
>
> Hi Kairui,
>
>> Just install `umount` and include the previous dracut pull requests we
>> opened, nothing else needed. I'm also testing this on fadump capable
>> machine, so far everything looks good.
>>
>
> I was trying to test it this afternoon. Ran into OOM issues with the
> above patch in capture kernel..
>
>>>
>>>
>>>>
>>>> #!/bin/sh
>>>> export PATH=/bin:/sbin
>>>>
>>>> [ -e /proc/self/mounts ] \
>>>> || (mkdir -p /proc && mount -t proc -o nosuid,noexec,nodev
proc /proc)
>>>>
>>>> grep -q '^sysfs /sys sysfs' /proc/self/mounts \
>>>> || (mkdir -p /sys && mount -t sysfs -o nosuid,noexec,nodev
sysfs /sys)
>>>>
>>>> if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>> mkdir /newroot
>>>> mount -t ramfs ramfs /newroot
>>>>
>>>> cp --reflink=auto --sparse=auto --preserve=mode,timestamps,links
>>>> -dfr /fadumproot/. /newroot/
>>>>
>>>> mkdir /newroot/sys
>>>> mkdir /newroot/proc
>>>> mkdir /newroot/oldroot
>>>> mount --move /sys /newroot/sys
>>>> mount --move /proc /newroot/proc
>>>>
>>>> cd /newroot || exit 1
>>>> pivot_root . oldroot
>>>>
>>>> # Clean up all mount points in oldroot
>>>> while true; do
>>>> loop=0
>>>> while read -r _ mp _; do
>>>> case $mp in
>>>> /oldroot/* )
>>>> umount "$mp" && loop=1
>>>> ;;
>>>> esac
>>>> done < /proc/mounts
>>>> [ $loop = 1 ] || break
>>>> done
>>>> umount -l oldroot
>>>>
>
>>>> # Jump to initramfs built for dump capture
>>>> exec /init
>
> Must have to do with the above. At the least, "exec /newroot/init".
> Also, isn't this missing a chroot..
Hi Hari,
Did you miss any pull request from dracut? Like:
https://github.com/dracutdevs/dracut/pull/1532
Sorry, I could have been clearer. When squash module is used on both
default and fadump initramfs, it is working but when the default
initramfs is built without squash enabled, running into the OOM
issues in capture kernel. Did you get a chance to try this scenario?
If yes, no issues in this scenario as well? fwiw, I see the below
trace in capture kernel when default initramfs is not built with
squash module but only "--add zz-fadumpinit":
[ 1.191970] Freeing unused kernel memory: 6528K
[ 1.191978] This architecture does not have kernel memory protection.
[ 1.191988] Run /init as init process
pivot_root: failed to change root from `.' to `oldroot': Invalid argument
/init: line 44: /proc/mounts: No such file or directory
umount: oldroot: umount failed: No such file or directory.
pivot_root: failed to change root from `.' to `oldroot': Invalid argument
/init: line 44: /proc/mounts: No such file or directory
umount: oldroot: umount failed: No such file or directory.
pivot_root: failed to change root from `.' to `oldroot': Invalid argument
/init: line 44: /proc/mounts: No such file or directory
umount: oldroot: umount failed: No such file or directory.
pivot_root: failed to change root from `.' to `oldroot': Invalid argument
/init: line 44: /proc/mounts: No such file or directory
umount: oldroot: umount failed: No such file or directory.
pivot_root: failed to change root from `.' to `oldroot': Invalid argument
/init: line 44: /proc/mounts: No such file or directory
umount: oldroot: umount failed: No such file or directory.
pivot_root: failed to change root from `.' to `oldroot': Invalid argument
/init: line 44: /proc/mounts: No such file or directory
umount: oldroot: umount failed: No such file or directory.
[ 1.389399] cp invoked oom-killer: gfp_mask=0x100cc2(GFP_HIGHUSER),
order=0, oom_score_adj=0
[ 1.389435] CPU: 6 PID: 424 Comm: cp Not tainted
5.12.10-300.fc34.ppc64le #1
[ 1.389443] Call Trace:
[ 1.389446] [c0000000199df310] [c0000000009e881c]
dump_stack+0xc0/0x104 (unreliable)
[ 1.389460] [c0000000199df360] [c0000000003d12a8] dump_header+0x64/0x284
[ 1.389469] [c0000000199df3f0] [c0000000003cf970]
oom_kill_process+0x2c0/0x2d0
[ 1.389477] [c0000000199df430] [c0000000003d0c14]
out_of_memory+0x224/0x780
[ 1.389485] [c0000000199df4d0] [c00000000045c7f0]
__alloc_pages_slowpath.constprop.0+0xc60/0xda0
[ 1.389494] [c0000000199df680] [c00000000045cc50]
__alloc_pages_nodemask+0x320/0x3d0
[ 1.389503] [c0000000199df710] [c00000000048b95c]
alloc_pages_current+0xcc/0x190
[ 1.389511] [c0000000199df760] [c0000000003c677c]
pagecache_get_page+0x36c/0x840
[ 1.389520] [c0000000199df840] [c0000000003c7454]
grab_cache_page_write_begin+0x34/0x70
[ 1.389529] [c0000000199df870] [c000000000548504]
simple_write_begin+0x44/0x100
[ 1.389537] [c0000000199df8b0] [c0000000003c17fc]
generic_perform_write+0xec/0x260
[ 1.389546] [c0000000199df950] [c0000000003c8d70]
__generic_file_write_iter+0x180/0x2f0
[ 1.389555] [c0000000199df9b0] [c0000000003c8f8c]
generic_file_write_iter+0xac/0x110
[ 1.389563] [c0000000199df9f0] [c0000000004face0]
do_iter_readv_writev+0x200/0x250
[ 1.389572] [c0000000199dfa50] [c0000000004fbf78]
do_iter_write+0xc8/0x270
[ 1.389579] [c0000000199dfaa0] [c00000000055aa68]
iter_file_splice_write+0x238/0x500
[ 1.389587] [c0000000199dfbb0] [c000000000559fac]
direct_splice_actor+0x4c/0x80
[ 1.389595] [c0000000199dfbd0] [c0000000005598e0]
splice_direct_to_actor+0x110/0x320
[ 1.389603] [c0000000199dfc50] [c000000000559b98]
do_splice_direct+0xa8/0x110
[ 1.389611] [c0000000199dfcc0] [c0000000004fe76c]
vfs_copy_file_range+0x1bc/0x5c0
[ 1.389619] [c0000000199dfd40] [c0000000004fec5c]
sys_copy_file_range+0xec/0x270
[ 1.389626] [c0000000199dfdb0] [c00000000002ddfc]
system_call_exception+0x10c/0x270
[ 1.389635] [c0000000199dfe10] [c00000000000d45c]
system_call_common+0xec/0x278
[ 1.389643] --- interrupt: c00 at 0x7fffa0ff6bc8
Thanks
Hari