On Mon, 28 Aug 2023 19:33:41 +0100 Paul Smith phhs80@gmail.com wrote:

On Mon, Aug 28, 2023 at 3:25 PM stan upaitag@zoho.com wrote:

...

...

Thanks, Stan. I guess the files are almost similar, since they have almost the same size:

I agree. Your issue isn't the issue I had. Debugging these early issues is hard, because it has to be done from the emergency console. But, I think I remember there being other consoles to switch to that had things like logs, and there are some commands available, if you do an ls /usr/bin or ls /usr/sbin you should be able to examine the system to some extent to find why it is failing. I remember using less and vi at least. Could you mount one of the partitions and save / cp or cat the logs to a file there, maybe in your home directory. Then you can examine them from a working system, and even attach them to a bugzilla or email.

...

# ls -n /boot/initramfs* -rw-------. 1 0 0 83396891 Jan 21 2020 /boot/initramfs-0-rescue-5cbe81aa795444b29a47ec1bf2b6dca1.img -rw-------. 1 0 0 37876860 Aug 14 11:45 /boot/initramfs-6.4.10-200.fc38.x86_64.img -rw-------. 1 0 0 37878341 Aug 26 12:28 /boot/initramfs-6.4.12-200.fc38.x86_64.img

Thanks, Stan. Finally, I was able to capture the log of

journalctl -xb

which is at:

https://bugzilla-attachments.redhat.com/attachment.cgi?id=1985687

So, this seems to be the error: Aug 28 19:05:39 localhost @ystemctl[626]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.

And it looks like it was triggered by this: Aug 28 19:05:39 localhost systemd[1]: Received SIGRTMIN+20 from PID 569 (plymouthd).

After that message, systemd starts shutting down instead of starting up.

Do you have fedora-release for fc38 installed? I can't see how you wouldn't, but just confirming. It looks like there is a bug in the code that is determining whether os-release is present, but only for warm restarts, not for cold boots. That would be plymouthd. Or whatever is creating the warm restart isn't creating it properly, so that plymouthd can recognize it, I think that would be systemd.

Were either of those in the large batch of updates you installed? If so, you could try downgrading them from koji to see if it fixes the error. https://koji.fedoraproject.org/koji/

On Tue, Aug 29, 2023 at 11:41 PM Paul Smith phhs80@gmail.com wrote:

...

...

...

...

Thanks, Stan. I guess the files are almost similar, since they have almost the same size:

I agree. Your issue isn't the issue I had. Debugging these early issues is hard, because it has to be done from the emergency console. But, I think I remember there being other consoles to switch to that had things like logs, and there are some commands available, if you do an ls /usr/bin or ls /usr/sbin you should be able to examine the system to some extent to find why it is failing. I remember using less and vi at least. Could you mount one of the partitions and save / cp or cat the logs to a file there, maybe in your home directory. Then you can examine them from a working system, and even attach them to a bugzilla or email.

...

# ls -n /boot/initramfs* -rw-------. 1 0 0 83396891 Jan 21 2020 /boot/initramfs-0-rescue-5cbe81aa795444b29a47ec1bf2b6dca1.img -rw-------. 1 0 0 37876860 Aug 14 11:45 /boot/initramfs-6.4.10-200.fc38.x86_64.img -rw-------. 1 0 0 37878341 Aug 26 12:28 /boot/initramfs-6.4.12-200.fc38.x86_64.img

Thanks, Stan. Finally, I was able to capture the log of

journalctl -xb

which is at:

https://bugzilla-attachments.redhat.com/attachment.cgi?id=1985687

So, this seems to be the error: Aug 28 19:05:39 localhost @ystemctl[626]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing.

And it looks like it was triggered by this: Aug 28 19:05:39 localhost systemd[1]: Received SIGRTMIN+20 from PID 569 (plymouthd).

After that message, systemd starts shutting down instead of starting up.

Do you have fedora-release for fc38 installed? I can't see how you wouldn't, but just confirming. It looks like there is a bug in the code that is determining whether os-release is present, but only for warm restarts, not for cold boots. That would be plymouthd. Or whatever is creating the warm restart isn't creating it properly, so that plymouthd can recognize it, I think that would be systemd.

Were either of those in the large batch of updates you installed? If so, you could try downgrading them from koji to see if it fixes the error. https://koji.fedoraproject.org/koji/

Thanks, Stan.

I do have:

# rpm -q fedora-release-38 fedora-release-38-36.noarch #

Moreover, both plytmouth and systemd that I have installed correspond to versions dated earlier than the date of the first occurrence of the reported issue.

With the older

kernel-6.4.10-200.fc38.x86_64

there is no issue.

Maybe the following is relevant:

---------------------------- # dracut --force --kver 6.4.12-200.fc38.x86_64 --verbose dracut: Executing: /usr/bin/dracut --force --kver 6.4.12-200.fc38.x86_64 --verbose dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found! dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found! dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found! dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found! dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found! dracut: *** Including module: bash *** dracut: *** Including module: systemd *** dracut: *** Including module: systemd-initrd *** dracut: *** Including module: systemd-sysusers *** dracut: *** Including module: nss-softokn *** dracut: *** Including module: dbus-broker *** dracut: *** Including module: rngd *** dracut: *** Including module: dbus *** dracut: *** Including module: i18n *** dracut: *** Including module: network-manager *** dracut: *** Including module: network *** dracut: *** Including module: ifcfg *** dracut: *** Including module: drm *** dracut: *** Including module: plymouth *** dracut: *** Including module: dm *** dracut: Skipping udev rule: 64-device-mapper.rules dracut: Skipping udev rule: 60-persistent-storage-dm.rules dracut: Skipping udev rule: 55-dm.rules dracut: *** Including module: kernel-modules *** dracut: *** Including module: kernel-modules-extra *** dracut: *** Including module: kernel-network-modules *** dracut: *** Including module: lvm *** dracut: Skipping udev rule: 64-device-mapper.rules dracut: Skipping udev rule: 56-lvm.rules dracut: Skipping udev rule: 60-persistent-storage-lvm.rules dracut: *** Including module: resume *** dracut: *** Including module: rootfs-block *** dracut: *** Including module: terminfo *** dracut: *** Including module: udev-rules *** dracut: Skipping udev rule: 40-redhat.rules dracut: Skipping udev rule: 50-firmware.rules dracut: Skipping udev rule: 50-udev.rules dracut: Skipping udev rule: 91-permissions.rules dracut: Skipping udev rule: 80-drivers-modprobe.rules dracut: Skipping udev rule: 70-persistent-net.rules dracut: *** Including module: dracut-systemd *** dracut: *** Including module: usrmount *** dracut: *** Including module: base *** dracut: *** Including module: fs-lib *** dracut: *** Including module: memstrack *** dracut: *** Including module: shutdown *** dracut: *** Including modules done *** dracut: *** Installing kernel module dependencies *** dracut: *** Installing kernel module dependencies done *** dracut: *** Resolving executable dependencies *** dracut: *** Resolving executable dependencies done *** dracut: *** Hardlinking files *** dracut: Mode: real dracut: Method: sha256 dracut: Files: 2153 dracut: Linked: 81 files dracut: Compared: 0 xattrs dracut: Compared: 974 files dracut: Saved: 563.14 KiB dracut: Duration: 0.057574 seconds dracut: *** Hardlinking files done *** dracut: *** Generating early-microcode cpio image *** dracut: *** Constructing AuthenticAMD.bin *** dracut: *** Store current command line parameters *** dracut: *** Stripping files *** dracut: *** Stripping files done *** dracut: *** Creating image file '/boot/initramfs-6.4.12-200.fc38.x86_64.img' *** dracut: Using auto-determined compression method 'pigz' dracut: *** Creating initramfs image file '/boot/initramfs-6.4.12-200.fc38.x86_64.img' done *** # -----------------------------------

Paul

On Wed, Aug 30, 2023 at 2:04 PM stan via users users@lists.fedoraproject.org wrote:

...

Maybe the following is relevant:

---

# dracut --force --kver 6.4.12-200.fc38.x86_64 --verbose dracut: Executing: /usr/bin/dracut --force --kver 6.4.12-200.fc38.x86_64 --verbose dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!

I see the connman errors when I build the initramfs, but it works fine.

...

dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found! dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found! dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found! dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found! dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!

I think these can be taken as warnings rather than errors, unless you are using busybox or network-wicked in the suspended version.

...

dracut: *** Including module: bash *** dracut: *** Including module: systemd *** dracut: *** Including module: systemd-initrd *** dracut: *** Including module: systemd-sysusers *** dracut: *** Including module: nss-softokn *** dracut: *** Including module: dbus-broker *** dracut: *** Including module: rngd *** dracut: *** Including module: dbus *** dracut: *** Including module: i18n *** dracut: *** Including module: network-manager *** dracut: *** Including module: network *** dracut: *** Including module: ifcfg *** dracut: *** Including module: drm *** dracut: *** Including module: plymouth *** dracut: *** Including module: dm *** dracut: Skipping udev rule: 64-device-mapper.rules dracut: Skipping udev rule: 60-persistent-storage-dm.rules dracut: Skipping udev rule: 55-dm.rules dracut: *** Including module: kernel-modules *** dracut: *** Including module: kernel-modules-extra *** dracut: *** Including module: kernel-network-modules *** dracut: *** Including module: lvm *** dracut: Skipping udev rule: 64-device-mapper.rules dracut: Skipping udev rule: 56-lvm.rules dracut: Skipping udev rule: 60-persistent-storage-lvm.rules dracut: *** Including module: resume *** dracut: *** Including module: rootfs-block *** dracut: *** Including module: terminfo *** dracut: *** Including module: udev-rules *** dracut: Skipping udev rule: 40-redhat.rules dracut: Skipping udev rule: 50-firmware.rules dracut: Skipping udev rule: 50-udev.rules dracut: Skipping udev rule: 91-permissions.rules dracut: Skipping udev rule: 80-drivers-modprobe.rules dracut: Skipping udev rule: 70-persistent-net.rules dracut: *** Including module: dracut-systemd *** dracut: *** Including module: usrmount *** dracut: *** Including module: base *** dracut: *** Including module: fs-lib *** dracut: *** Including module: memstrack *** dracut: *** Including module: shutdown *** dracut: *** Including modules done *** dracut: *** Installing kernel module dependencies *** dracut: *** Installing kernel module dependencies done *** dracut: *** Resolving executable dependencies *** dracut: *** Resolving executable dependencies done *** dracut: *** Hardlinking files *** dracut: Mode: real dracut: Method: sha256 dracut: Files: 2153 dracut: Linked: 81 files dracut: Compared: 0 xattrs dracut: Compared: 974 files dracut: Saved: 563.14 KiB dracut: Duration: 0.057574 seconds dracut: *** Hardlinking files done *** dracut: *** Generating early-microcode cpio image *** dracut: *** Constructing AuthenticAMD.bin *** dracut: *** Store current command line parameters *** dracut: *** Stripping files *** dracut: *** Stripping files done *** dracut: *** Creating image file '/boot/initramfs-6.4.12-200.fc38.x86_64.img' *** dracut: Using auto-determined compression method 'pigz' dracut: *** Creating initramfs image file '/boot/initramfs-6.4.12-200.fc38.x86_64.img' done *** #

This all looks fine. The journalctl log you posted showed that the initramfs was working properly, and was able to start systemd to do the actual boot. It only failed at verifying that the os you were trying to start was legitimate.

Given the fact that an older kernel works, but a newer one doesn't, it sure seems like it is somehow related to the kernel. Since the same version of systemd works on an old kernel, it is unlikely to be systemd. Maybe you should change the bugzilla to point to the kernel. You could also look through the kernel updates at https://lkml.org/ to see if there is one related to warm restarts. I think that might also be called suspend. There are lot of updates, though, so this is a significant task.

Thanks, again, Stan. I will be paying attention to

https://lkml.org/

Paul

On Thu, Aug 31, 2023 at 3:50 PM George N. White III gnwiii@gmail.com wrote:

...

...

[...] Given the fact that an older kernel works, but a newer one doesn't, it sure seems like it is somehow related to the kernel. Since the same version of systemd works on an old kernel, it is unlikely to be systemd. Maybe you should change the bugzilla to point to the kernel. You could also look through the kernel updates at https://lkml.org/ to see if there is one related to warm restarts. I think that might also be called suspend. There are lot of updates, though, so this is a significant task.

Thanks, again, Stan. I will be paying attention to

https://lkml.org/

Search for "site: lkml.org warm reboot restart" shows a lot of activity, often related to specific hardware, so it is most useful if you know low level details for your hardware. You may want to try running a stripped down configuration to see of some add-on card causing the issue.

Thanks, George, for your suggestion!

Fortunately, the bug report I have filed has recently had reasonable activity.

Paul