On Thu, Jun 17, 2021 at 10:08 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
On 16/06/21 1:26 pm, Kairui Song wrote:
> On Wed, Jun 16, 2021 at 2:32 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
>> On 16/06/21 12:42 am, Kairui Song wrote:
>>> On Tue, Jun 15, 2021 at 2:36 PM Hari Bathini <hbathini(a)linux.ibm.com>
wrote:
>>>>
>>>>
>>>>
>>>> On 14/06/21 3:59 pm, Hari Bathini wrote:
>>>>>
>>>>>
>>>>> On 10/06/21 8:29 pm, Hari Bathini wrote:
>>>>>> In case of fadump, the initramfs image has to be built to boot
into
>>>>>> the production environment as well as to offload the active
crash dump
>>>>>> to the specified dump target (for boot after crash). As the same
image
>>>>>> would be used for both boot scenarios, it could not be built
optimally
>>>>>> while accommodating both cases.
>>>>>>
>>>>>> Use --include to include the initramfs image built for
offloading
>>>>>> active crash dump to the specified dump target. Also, introduce
a new
>>>>>> out-of-tree dracut module (99zz-fadumpinit) that installs a
customized
>>>>>> init program while moving the default /init to /init.dracut.
This
>>>>>> customized init program is leveraged to isolate fadump image
within
>>>>>> the default initramfs image by kicking off default boot process
>>>>>> (exec /init.dracut) for regular boot scenario and activating
fadump
>>>>>> initramfs image, if the system is booting after a crash.
>>>>>>
>>>>>> Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
>>>>>> ---
>>>>>>
>>>>>> * Applying this patch, the initramfs image built with fadump
support
>>>>>> can be too large, without
>>>>>>
https://github.com/dracutdevs/dracut/pull/1532
>>>>>>
>>>>>> TODO:
>>>>>> * Fix missing ssh identify file /root/.ssh/kdump_id_rsa with
>>>>>> squash not used and remote (ssh) dump target configured.
>>>>>> Changes in V3:
>>>>>> * Removed dependency with squash module by extracting fadump
>>>>>> initramfs to /fadumproot directory.
>>>>>> * Moved code that builds initramfs for fadump to mkfadumprd and
>>>>>> added cleanup support based on inputs from Kairui.
>>>>>>
>>>>>> Changes in V2:
>>>>>> * Moved kernel modules installation from install() to
installkernel() for
>>>>>> zz-fadumpinit dracut module.
>>>>>> * Updated kexec-tools.spec to install zz-fadumpinit module
files
>>>>>> appropriately.
>>>>>> * Fixed error check for capture initramfs image isolation with
dracut.
>>>>>>
>>>>>>
>>>>>> dracut-fadump-init-fadump.sh | 20 ++++++++++++++
>>>>>> dracut-fadump-module-setup.sh | 17 ++++++++++++
>>>>>> dracut-module-setup.sh | 6 +---
>>>>>> kdumpctl | 22 +++-------------
>>>>>> kexec-tools.spec | 14 ++++++++++
>>>>>> mkfadumprd | 58
>>>>>> +++++++++++++++++++++++++++++++++++++++++
>>>>>> 6 files changed, 115 insertions(+), 22 deletions(-)
>>>>>> create mode 100644 dracut-fadump-init-fadump.sh
>>>>>> create mode 100644 dracut-fadump-module-setup.sh
>>>>>> create mode 100644 mkfadumprd
>>>>>>
>>>>>> diff --git a/dracut-fadump-init-fadump.sh
b/dracut-fadump-init-fadump.sh
>>>>>> new file mode 100644
>>>>>> index 0000000..736a7dd
>>>>>> --- /dev/null
>>>>>> +++ b/dracut-fadump-init-fadump.sh
>>>>>> @@ -0,0 +1,20 @@
>>>>>> +#!/bin/sh
>>>>>> +PATH=/bin:/sbin
>>>>>> +
>>>>>> +[ -e /proc/self/mounts ] \
>>>>>> + || (mkdir -p /proc && mount -t proc -o
nosuid,noexec,nodev proc
>>>>>> /proc)
>>>>>> +
>>>>>> +grep -q '^sysfs /sys sysfs' /proc/self/mounts \
>>>>>> + || (mkdir -p /sys && mount -t sysfs -o
nosuid,noexec,nodev sysfs
>>>>>> /sys)
>>>>>> +
>>>>>> +# If system is booting after a crash, switch to initramfs built
for
>>>>>> dump capture
>>>>>> +if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
>>>>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>>>>> + mkdir -p /newroot
>>>>>> + mount -t ramfs ramfs /newroot
>>>>
>>>>
>>>> Hi Kairui,
>>>>
>>>>>> + mv /fadumproot/* /newroot
>>>>
>>>>> Similar to the dracut --include issue, this would also leave behind
>>>>> filenames beginning with '.'. As replacing '*' with
'.' won't work,
>>>>
>>>> How about the below instead:
>>>>
>>>> for FILE in $(ls -A fadumproot); do mv fadumproot/$FILE newroot;
done
>>>>
>>>> Need to install 'ls' while setting up the module though..
>>>>
>>>
>>> Hi Hari,
>>
>> Hi Kairui,
>>
>>>
>>> After testing with squash module enabled for both initramfs and the
>>> embedded fadump initramfs, I think your previous idea that use `cp`
>>> consumes same amount of memory as with `mv` is right.
>>> Using `mv` will only save memory when the normal initramfs is not
>>> using squash module, but I believe we will have to enable squash
>>> module for both initramfs to save memory for fadump, am I right?
>>
>> It would be better if we can stick with the same build parameters for
>> the default initramfs except for "--add zz-fadumpinit". So, the way
>> I look at it, using squash module to build default initramfs image
>> must depend on whether it was already with squash or not. Let me know
>> your thoughts..
>
> I recommended both initramfs to use squash image, this will help to
> minimize the memory usage for fadump, so we can use same crashkernel=
> value for both fadump and kdump.
> In my test, when both initramfs uses squash module, fadump works with
> 384M of memory, Without squash module enabled for both initramfs,
> seems have to reserve 512M of memory. No error or warning message
> observed. (just a umount failure, which happens once in the umount
> loop in init script, that's expected, we can silence that later)
>
>>
>>>
>>> And base on that assumption (both initramfs uses squash module), I
>>> made a scratch `init` file for fadump setup, this one should work fine
>>> and consumes less memory, simply call `switch_root` will left the old
>>> mount points uncleared so the old squash img is never dropped and
>>> consumes ~100m of memory. How do you think? Other boot init program
>>> will cleanup old mount points recursively (eg. systemd), so I think it
>>> make sense for this script to do something similar.
>>> (this require 99squash module to install umount as well, and
>>> 99zz-fadump will need to install pivot_root)
>>
>> I would like to test this. Other than installing umount with squash
>> module, any other changes needed (say in init-squash.sh)?
>
Hi Kairui,
> Just install `umount` and include the previous dracut pull requests we
> opened, nothing else needed. I'm also testing this on fadump capable
> machine, so far everything looks good.
>
I was trying to test it this afternoon. Ran into OOM issues with the
above patch in capture kernel..
>>
>>
>>>
>>> #!/bin/sh
>>> export PATH=/bin:/sbin
>>>
>>> [ -e /proc/self/mounts ] \
>>> || (mkdir -p /proc && mount -t proc -o nosuid,noexec,nodev
proc /proc)
>>>
>>> grep -q '^sysfs /sys sysfs' /proc/self/mounts \
>>> || (mkdir -p /sys && mount -t sysfs -o nosuid,noexec,nodev
sysfs /sys)
>>>
>>> if [ -f /proc/device-tree/rtas/ibm,kernel-dump ] || [ -f
>>> /proc/device-tree/ibm,opal/dump/mpipl-boot ]; then
>>> mkdir /newroot
>>> mount -t ramfs ramfs /newroot
>>>
>>> cp --reflink=auto --sparse=auto --preserve=mode,timestamps,links
>>> -dfr /fadumproot/. /newroot/
>>>
>>> mkdir /newroot/sys
>>> mkdir /newroot/proc
>>> mkdir /newroot/oldroot
>>> mount --move /sys /newroot/sys
>>> mount --move /proc /newroot/proc
>>>
>>> cd /newroot || exit 1
>>> pivot_root . oldroot
>>>
>>> # Clean up all mount points in oldroot
>>> while true; do
>>> loop=0
>>> while read -r _ mp _; do
>>> case $mp in
>>> /oldroot/* )
>>> umount "$mp" && loop=1
>>> ;;
>>> esac
>>> done < /proc/mounts
>>> [ $loop = 1 ] || break
>>> done
>>> umount -l oldroot
>>>
>>> # Jump to initramfs built for dump capture
>>> exec /init
Must have to do with the above. At the least, "exec /newroot/init".
Also, isn't this missing a chroot..
With above PR merged, it work well with crashkernel=384M during my
test. If it still fails with OOM, can you share me your initramfs?
It's strange that I actually seeing this consuming much less memory.
And in the patch, it's using pivot_root, so no need to chroot.