On Mon, Jun 17, 2019 at 7:13 PM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
On 14/06/19 1:16 PM, Kairui Song wrote:
On Wed, Jun 12, 2019 at 2:01 AM Hari Bathini <hbathini(a)linux.ibm.com> wrote:
In FADump mode, when the dump targer is '/home', regular boot is failing
with logs like below:
Starting Reload Configuration from the Real Root...
[ 3.289716] EXT4-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null)
[FAILED] Failed to mount /kdumproot/home.
See 'systemctl status kdumproot-home.mount' for details.
[DEPEND] Dependency failed for Local File Systems.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut initqueue hook.
Starting Kdump Emergency...
[ OK ] Stopped dracut cmdline hook.
[ OK ] Stopped dracut ask for additional cmdline parameters.
Starting Setup Virtual Console...
Starting Create Volatile Files and Directories...
[ OK ] Started Cleanup squashfs mounts when switch root.
[ OK ] Started Kdump Emergency.
and eventually ending up in emergency mode. 'kdumproot/home' seems to
be mounted successfully though. 'Reload from real root' is altering the
dependency graph leading to this mess. Updated kdump-capture.service
unit file to run before initrd-parse-etc.service to avoid running into
such problems. While here, dropped the deprecated 'OnFailureIsolate'
in favour of 'OnFailureJobMode'. Tested this change successfully in both
KDump and FADump modes with different dump methods (local, ssh, nfs).
Signed-off-by: Hari Bathini <hbathini(a)linux.ibm.com>
---
dracut-kdump-capture.service | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/dracut-kdump-capture.service b/dracut-kdump-capture.service
index 57139c9..df4a437 100644
--- a/dracut-kdump-capture.service
+++ b/dracut-kdump-capture.service
@@ -7,12 +7,11 @@
[Unit]
Description=Kdump Vmcore Save Service
-After=initrd.target initrd-parse-etc.service sysroot.mount
-After=dracut-initqueue.service dracut-pre-mount.service dracut-mount.service
dracut-pre-pivot.service
-Before=initrd-cleanup.service
+After=initrd-root-fs.target remote-fs.target
+Before=initrd-parse-etc.service
ConditionPathExists=/etc/initrd-release
OnFailure=emergency.target
-OnFailureIsolate=yes
+OnFailureJobMode=isolate
[Service]
Environment=DRACUT_SYSTEMD=1
Hi Hari,
Hi Kairui,
Thanks for the review.
I still don't understand how this fixed the issue, and I can't
repdocue it eigher (by generating a fadump initrd on my VM).
All I can say is, I faced issues with mount and initrd-parse-etc.service previously as
well.
FWIW, I have seen failure on a system with the below configuration:
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 4M 0 part
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 99G 0 part
├─rhel_ltczep2--lp1-root 253:0 0 50G 0 lvm /
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdb 8:16 0 50G 0 disk
└─sdb1 8:17 0 50G 0 part
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdc 8:32 0 50G 0 disk
└─sdc1 8:33 0 50G 0 part
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
sdd 8:48 1 14.4G 0 disk
└─sdd1 8:49 1 14.4G 0 part
├─rhel_ltczep2--lp1-swap 253:1 0 5G 0 lvm [SWAP]
└─rhel_ltczep2--lp1-home 253:2 0 158.4G 0 lvm /home
System is hung when "systemd.journald.forward_to_console=1" parameter is
passed
to the kernel. So, attaching the console log on the failing system with parameters
"systemd.log_target=console systemd.log_level=debug" passed to the kernel...
The VM I use is a default install which use LVM, not sure if this is
related. /home is on a dedicated disk. I did found some other issue
with fadump initrd with /home as target:
[FAILED] Failed unmounting /kdumproot/home.
[ OK ] Stopped File System Check on /dev/mapper/rhel-swap.
[ OK ] Stopped udev Kernel Device Manager.
[ OK ] Stopped dracut pre-udev hook.
[ OK ] Stopped dracut cmdline hook.
But then everything goes on well.
It give more log with "systemd.log_target=console
systemd.journald.forward_to_console=1" appended to kernel cmdline:
kdumproot-home.mount: Failed to execute command: No such file or
directory
kdumproot-home.mount: Failed at step EXEC spawning /usr/bin/umount: No
such file or directory
And could be fixed by:
diff --git a/dracut-module-setup.sh b/dracut-module-setup.sh
index 2998c72..5222040 100755
--- a/dracut-module-setup.sh
+++ b/dracut-module-setup.sh
@@ -30,7 +30,7 @@ depends() {
done
}
- if is_squash_available; then
+ if is_squash_available && ! is_fadump_capable; then
_dep="$_dep squash"
else
dwarning "Required modules to build a squashed kdump image is
missing!"
This patch helps with the unmount issue (seen even with the patch I proposed)..
- Hari
Thanks for the update! If I get it right, this seems a known systemd bug:
https://github.com/systemd/systemd/issues/10872
As I did see following logs in the attachment:
dev-mapper-rhel_ltczep2\x2d\x2dlp1\x2dhome.device: Failed to send unit
change signal for dev-mapper-rhel_ltczep2\x2d\x2dlp1\x2dhome.device:
Connection reset by peer
Child 981 (mount) died (code=exited, status=0/SUCCESS)
kdumproot-home.mount: Child 981 belongs to kdumproot-home.mount.
kdumproot-home.mount: Mount process exited, code=exited status=0
kdumproot-home.mount: Mount process finished, but there is no mount.
kdumproot-home.mount: Failed with result 'protocol'.
kdumproot-home.mount: Changed mounting -> failed
kdumproot-home.mount: Job kdumproot-home.mount/start finished, result=failed
I'll try again to see if I can reproduce it...
--
Best Regards,
Kairui Song