On Mon, 2023-08-21 at 17:13 +0100, Paul Smith wrote:

On Mon, Aug 21, 2023 at 5:10 PM Patrick O'Callaghan pocallaghan@gmail.com wrote:

...

...

Are other also experience that with the new kernel:

6.4.10-200.fc38.x86_64

I have already file a bug but no answer:

https://bugzilla.redhat.com/show_bug.cgi?id=2232838

I didn't have any problems, but I updated to 6.4.11 this morning so maybe try that.

Thanks, Patrick: My bad: the problematic kernel is:

kernel-6.4.11-200.fc38.x86_64

That's the one I'm using now. Haven't seen any issues with it, touch wood.

poc

On Tue, Aug 22, 2023 at 2:26 PM Ranjan Maitra mlmaitra@gmx.com wrote:

...

...

...

Are other also experience that with the new kernel:

6.4.10-200.fc38.x86_64

I have already file a bug but no answer:

https://bugzilla.redhat.com/show_bug.cgi?id=2232838

I didn't have any problems, but I updated to 6.4.11 this morning so maybe try that.

Thanks, Patrick: My bad: the problematic kernel is:

kernel-6.4.11-200.fc38.x86_64

Can you boot into the older kernel?

Yes, Ranjan, I can boot into an older kernel. Thanks!

Paul

On Thu, Aug 24, 2023 at 4:26 PM Ranjan Maitra mlmaitra@gmx.com wrote:

...

...

...

...

...

Are other also experience that with the new kernel:

6.4.10-200.fc38.x86_64

I have already file a bug but no answer:

https://bugzilla.redhat.com/show_bug.cgi?id=2232838

I didn't have any problems, but I updated to 6.4.11 this morning so maybe try that.

Thanks, Patrick: My bad: the problematic kernel is:

kernel-6.4.11-200.fc38.x86_64

Can you boot into the older kernel?

Yes, Ranjan, I can boot into an older kernel. Thanks!

When such a thing very rarely has happened to me, it usually means that grub did not finish installing the new kernel update. Try reinstalling perhaps? Or rebuild grub?

Thanks, Rajan and old sixpack13. I have tried to remove and then install the problematic kernel, but that does not fix the problem, unfortunately.

I have been able to take a screen-shot:

https://i.imgur.com/nAsE6i6.jpg

Paul

On 8/24/23 09:44, Paul Smith wrote:

On Thu, Aug 24, 2023 at 4:26 PM Ranjan Maitra mlmaitra@gmx.com wrote:

...

...

...

...

...

> Are other also experience that with the new kernel: > > 6.4.10-200.fc38.x86_64 > > I have already file a bug but no answer: > > https://bugzilla.redhat.com/show_bug.cgi?id=2232838

I didn't have any problems, but I updated to 6.4.11 this morning so maybe try that.

Thanks, Patrick: My bad: the problematic kernel is:

kernel-6.4.11-200.fc38.x86_64

Can you boot into the older kernel?

Yes, Ranjan, I can boot into an older kernel. Thanks!

When such a thing very rarely has happened to me, it usually means that grub did not finish installing the new kernel update. Try reinstalling perhaps? Or rebuild grub?

Thanks, Rajan and old sixpack13. I have tried to remove and then install the problematic kernel, but that does not fix the problem, unfortunately.

I have been able to take a screen-shot:

https://i.imgur.com/nAsE6i6.jpg

If you remove the "quiet rhgb" from the end of the grub line before booting, you might be able to see better what's failing at the end.

On Fri, Aug 25, 2023 at 4:23 PM stan via users users@lists.fedoraproject.org wrote:

...

I have been able to take a screen-shot:

https://i.imgur.com/nAsE6i6.jpg

This means that dracut was unable to start systemd in order to continue the boot once the initramfs was finished creating the bootstrap. I have had this happen, but not with stock kernels. I build custom kernels, and dracut was not building a complete initramfs when they were installed. I had to write a script that created a custom file that told dracut to include all missing libraries, and run dracut again after install to fix the problem. The missing libraries included some vital systemd libraries. Because library versions change, I run the script to be sure that the custom file includes the latest libraries. If this *is* the cause, it will be obvious in /boot. The failing initramfs size will be about half the size of an initramfs file that works. I could not find any reason that dracut wasn't working properly when it first installed the kernel; all the settings said it should have put those libraries in, but I just could not get it to do so.

If it is your issue, post back and I'll attach the file of libraries and the directory to put it in, as well as the dracut command to run. They might not work for you, since the version has to be included. There is also a way to actually examine what is in the initramfs, so you could see if the systemd libraries were there. From the dracut man page,

Inspecting the Contents To see the contents of the image created by dracut, you can use the lsinitrd tool.

       # lsinitrd | less

   To display the contents of a file in the initramfs also use the

lsinitrd tool:

       # lsinitrd -f /etc/ld.so.conf
       include ld.so.conf.d/*.conf

Thanks to all for your help!

I have meanwhile been able to take photos from the

journalctl -xb

output, and I think that now the cause of the problem is isolated.

I was hopeful that the new kernel update would fix the problem, but it did not.

The photos of the journalctl logs are at:

https://i.imgur.com/NJAjOmN.jpg

Paul

On Sat, Aug 26, 2023 at 3:30 PM stan via users users@lists.fedoraproject.org wrote:

...

...

...

I have been able to take a screen-shot:

https://i.imgur.com/nAsE6i6.jpg

This means that dracut was unable to start systemd in order to continue the boot once the initramfs was finished creating the bootstrap. I have had this happen, but not with stock kernels. I build custom kernels, and dracut was not building a complete initramfs when they were installed. I had to write a script that created a custom file that told dracut to include all missing libraries, and run dracut again after install to fix the problem. The missing libraries included some vital systemd libraries. Because library versions change, I run the script to be sure that the custom file includes the latest libraries. If this *is* the cause, it will be obvious in /boot. The failing initramfs size will be about half the size of an initramfs file that works. I could not find any reason that dracut wasn't working properly when it first installed the kernel; all the settings said it should have put those libraries in, but I just could not get it to do so.

If it is your issue, post back and I'll attach the file of libraries and the directory to put it in, as well as the dracut command to run. They might not work for you, since the version has to be included. There is also a way to actually examine what is in the initramfs, so you could see if the systemd libraries were there. From the dracut man page,

Inspecting the Contents To see the contents of the image created by dracut, you can use the lsinitrd tool.

       # lsinitrd | less

   To display the contents of a file in the initramfs also use

the lsinitrd tool:

       # lsinitrd -f /etc/ld.so.conf
       include ld.so.conf.d/*.conf

Thanks to all for your help!

I have meanwhile been able to take photos from the

journalctl -xb

output, and I think that now the cause of the problem is isolated.

I was hopeful that the new kernel update would fix the problem, but it did not.

The photos of the journalctl logs are at:

https://i.imgur.com/NJAjOmN.jpg

You didn't show the listing of the /boot directory.

ls -n /boot/ or ls -n /boot/initramfs*

Is the initramfs for the failing kernel smaller than the initramfs for the successful kernel? If you can still boot into an older kernel, you can try rebuilding the initramfs manually, to see if it will fix any problem. You have to run this within the /boot directory, as root or sudo. Your command should be something like the following.

# /usr/bin/dracut -f -v --no-compress --no-uefi initramfs-6.4.11-200.fc38.x86_64.img --kver 6.4.11-200.fc38.x86_64

I use no compression and no uefi, but you can remove those if they don't fit your system.

Once you have a new initramfs, run lsinitrd on it and redirect it into a file. Do the same for a working initramfs. Then run a diff on the two with the output piped to less. For example,

lsinitrd /boot/initramfs-6.4.11-200.fc38.x86_64.img > new_initramfs.txt lsinitrd /boot/initramfs-6.4.8-200.fc38.x86_64.img > old_initramfs.txt diff old_initramfs.txt new_initramfs.txt | less

There should be almost no differences if the initramfs is OK.

Thanks, Stan. I guess the files are almost similar, since they have almost the same size:

# ls -n /boot/initramfs* -rw-------. 1 0 0 83396891 Jan 21 2020 /boot/initramfs-0-rescue-5cbe81aa795444b29a47ec1bf2b6dca1.img -rw-------. 1 0 0 37876860 Aug 14 11:45 /boot/initramfs-6.4.10-200.fc38.x86_64.img -rw-------. 1 0 0 37878341 Aug 26 12:28 /boot/initramfs-6.4.12-200.fc38.x86_64.img #

Paul

On Mon, 21 Aug 2023 20:30:44 -0700 Samuel Sieb samuel@sieb.net wrote:

On 8/21/23 08:56, Paul Smith wrote:

...

Are other also experience that with the new kernel:

6.4.10-200.fc38.x86_64

I have already file a bug but no answer:

https://bugzilla.redhat.com/show_bug.cgi?id=2232838

You didn't really provide any info that would help. Can you get any journal output or the sos file? Is there any indication of what is going wrong?

FWIW .... I had a similar problem recently with a F37 system. After being unable to figure out a reason for the emergency mode -- in particular journalctl was clean, as was fstab -- I re-installed, which resolved the problem.

On 8/22/23 03:37, Paul Smith wrote:

On Tue, Aug 22, 2023 at 4:31 AM Samuel Sieb samuel@sieb.net wrote:

...

...

Are other also experience that with the new kernel:

6.4.10-200.fc38.x86_64

I have already file a bug but no answer:

https://bugzilla.redhat.com/show_bug.cgi?id=2232838

You didn't really provide any info that would help. Can you get any journal output or the sos file? Is there any indication of what is going wrong?

Thanks, Samuel and Geoffrey.

I have meanwhile uploaded the log file corresponding to

journalctl -xb

That's not the log from the failed boot. It looks like you booted to the previous kernel and ran that from the running system. The log won't be written to disk, so you have to look at it right then and possibly save it to some external storage if possible.