Sorry for this rather ill-defined bug report, but I'm hoping someone will recognize the symptoms or at least suggest a way to debug this.
Ever since Fedora 33 was released we've been unable to produce a working virt-builder template. The template that we have made fails at switch root.
template: https://builder.libguestfs.org/fedora-33.xz bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1911177
An additional problem is I cannot find any way to debug the problem. Something (systemd? dracut?) asks for a root password in order to get to the emergency shell, and I can find no way around this nor any password which works. I wish there was a "stop asking for a password" option on the kernel command line.
We use the same process to make these as we have always used, this kickstart:
https://github.com/libguestfs/libguestfs/blob/master/builder/templates/fedor...
is fed to this virt-install script:
https://github.com/libguestfs/libguestfs/blob/master/builder/templates/fedor...
and the result is minimized. (I tried turning off the minimization steps, it seems the output of virt-install + ks is broken, not anything we do afterwards).
Rich.
On Wed, Jan 13, 2021 at 10:16:43AM +0000, Richard W.M. Jones wrote:
Sorry for this rather ill-defined bug report, but I'm hoping someone will recognize the symptoms or at least suggest a way to debug this.
Ever since Fedora 33 was released we've been unable to produce a working virt-builder template. The template that we have made fails at switch root.
template: https://builder.libguestfs.org/fedora-33.xz bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1911177
An additional problem is I cannot find any way to debug the problem.
In this image, initrd-switch-root.service fails because "/sysroot does not contain /etc/os-release". This is visible when you boot with "systemd.log_level=debug systemd.journald.forward_to_console=1" as described on https://freedesktop.org/wiki/Software/systemd/Debugging/
Something (systemd? dracut?) asks for a root password in order to get to the emergency shell, and I can find no way around this nor any password which works. I wish there was a "stop asking for a password" option on the kernel command line.
This one is weird. I tried "rd.shell" and "rd.break", which should just drop to dracut shell, but it didn't work.
BTW, partition setup looks weird, too. vda1 is 1MB empty partition, then vda2 is /boot and vda3 is rootfs. Is this how we organise disks by default?
On Wed, Jan 13, 2021 at 12:02:33PM +0100, Tomasz Torcz wrote:
On Wed, Jan 13, 2021 at 10:16:43AM +0000, Richard W.M. Jones wrote:
Sorry for this rather ill-defined bug report, but I'm hoping someone will recognize the symptoms or at least suggest a way to debug this.
Ever since Fedora 33 was released we've been unable to produce a working virt-builder template. The template that we have made fails at switch root.
template: https://builder.libguestfs.org/fedora-33.xz bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1911177
An additional problem is I cannot find any way to debug the problem.
In this image, initrd-switch-root.service fails because "/sysroot does not contain /etc/os-release". This is visible when you boot with "systemd.log_level=debug systemd.journald.forward_to_console=1" as described on https://freedesktop.org/wiki/Software/systemd/Debugging/
Interesting thanks - I can reproduce the error using your suggested command line:
[ 3.167930] systemctl[272]: Failed to switch root: Specified switch root path '/sysroot' does not seem to be an OS tree. os-release file is missing. [ 3.174123] systemd[1]: initrd-switch-root.service: Main process exited, code=exited, status=1/FAILURE [ 3.176615] systemd[1]: initrd-switch-root.service: Failed with result 'exit-code'. [ 3.179342] systemd[1]: Failed to start Switch Root. [FAILED] Failed to start Switch Root.
I wonder what causes it to say that? The disk image itself has /etc/os-release -> ../usr/lib/os-release and /usr/lib/os-release is a regular file. The symlink ought to work since it is a relative path.
The kernel finds the virtio disk so it has at least the block drivers.
Something (systemd? dracut?) asks for a root password in order to get to the emergency shell, and I can find no way around this nor any password which works. I wish there was a "stop asking for a password" option on the kernel command line.
This one is weird. I tried "rd.shell" and "rd.break", which should just drop to dracut shell, but it didn't work.
BTW, partition setup looks weird, too. vda1 is 1MB empty partition, then vda2 is /boot and vda3 is rootfs. Is this how we organise disks by default?
AFAIK we don't do anything unusual. We're using autopart --type=plain in the kickstart which should make a non-LVM partition. I suppose vda1 must have something to do with UEFI.
Thanks anyway, this gets us a little bit further.
Rich.
-- Tomasz .. oo o. oo o. .o .o o. o. oo o. .. Torcz .. .o .o .o .o oo oo .o .. .. oo oo o.o.o. .o .. o. o. o. o. o. o. oo .. .. o. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Jan 13, 2021 at 10:16:43AM +0000, Richard W.M. Jones wrote:
Sorry for this rather ill-defined bug report, but I'm hoping someone will recognize the symptoms or at least suggest a way to debug this.
Ever since Fedora 33 was released we've been unable to produce a working virt-builder template. The template that we have made fails at switch root.
template: https://builder.libguestfs.org/fedora-33.xz bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1911177
An additional problem is I cannot find any way to debug the problem.
I would probably suggest to run an interactive anaconda install that reasonably closely matches the options you desire. This should give a working installed image that contains a corresponding working /root/install.ks file.
Compare that to the ks virt-builder is using and bisect the differences.
Something (systemd? dracut?) asks for a root password in order to get to the emergency shell, and I can find no way around this nor any password which works. I wish there was a "stop asking for a password" option on the kernel command line.
We use the same process to make these as we have always used, this kickstart:
https://github.com/libguestfs/libguestfs/blob/master/builder/templates/fedor...
Have you tried removing the whole of the %post script there ? The main ks rules that all look sane enough. It feels like that dracut invokation in %post is most likely thing to be causing a problem. Or perhaps the args to the 'bootloader' statement need an change in some manner ?
Regards, Daniel
On Wed, Jan 13, 2021 at 11:34:09AM +0000, Daniel P. Berrangé wrote:
On Wed, Jan 13, 2021 at 10:16:43AM +0000, Richard W.M. Jones wrote:
We use the same process to make these as we have always used, this kickstart:
https://github.com/libguestfs/libguestfs/blob/master/builder/templates/fedor...
Have you tried removing the whole of the %post script there ? The main ks rules that all look sane enough. It feels like that dracut invokation in %post is most likely thing to be causing a problem. Or perhaps the args to the 'bootloader' statement need an change in some manner ?
Weirdly it works when I comment out the dnf update:
%post # Ensure the installation is up-to-date. -dnf -y --best upgrade +#dnf -y --best upgrade
which I certainly wasn't expecting. At least I can ship a working image now.
Rich.
Hi,
An additional problem is I cannot find any way to debug the problem. Something (systemd? dracut?) asks for a root password in order to get to the emergency shell, and I can find no way around this nor any password which works. I wish there was a "stop asking for a password" option on the kernel command line.
Trapped into the same thing a few days ago. dracut uses sulogin now.
Run dracut with --no-hostonly. Adding dracut-config-generic to the kickstart package list should have the same effect (not tested though). That'll disable the root login in the initramfs and sulogin gives you a shell when you tap enter.
HTH, Gerd