https://bugzilla.redhat.com/show_bug.cgi?id=1430297
This has been going on for a few weeks with no attention. It prevents anyone from running libguestfs on Fedora 26.
It's (IMHO) pretty serious for anyone using virtualization: "kernel doesn't boot on qemu" should be a blocker for a Fedora release.
Can anyone see what's going on?
Rich.
On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=1430297
This has been going on for a few weeks with no attention. It prevents anyone from running libguestfs on Fedora 26.
It's (IMHO) pretty serious for anyone using virtualization: "kernel doesn't boot on qemu" should be a blocker for a Fedora release.
Can anyone see what's going on?
The earlier case looks like https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed in rc2. I'm not sure about the latter case, though Thorsten Leemhuis reported intermittent crash / hang during boot with rc2 as:
https://bugzilla.kernel.org/show_bug.cgi?id=194911
Since rc2 I've seen occasional failures which are probably the same as the ones Thorsten was seeing, but usually it works OK. I run 26-on-26 VMs via virt-manager quite a lot, they certainly don't fail every time.
On Tue, Mar 21, 2017 at 12:58:06PM -0700, Adam Williamson wrote:
On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=1430297
This has been going on for a few weeks with no attention. It prevents anyone from running libguestfs on Fedora 26.
It's (IMHO) pretty serious for anyone using virtualization: "kernel doesn't boot on qemu" should be a blocker for a Fedora release.
Can anyone see what's going on?
The earlier case looks like https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed in rc2. I'm not sure about the latter case, though Thorsten Leemhuis reported intermittent crash / hang during boot with rc2 as:
https://bugzilla.kernel.org/show_bug.cgi?id=194911
Since rc2 I've seen occasional failures which are probably the same as the ones Thorsten was seeing, but usually it works OK. I run 26-on-26 VMs via virt-manager quite a lot, they certainly don't fail every time.
Does libguestfs-test-tool work for you in F26?
Rich.
On Tue, 2017-03-21 at 20:48 +0000, Richard W.M. Jones wrote:
On Tue, Mar 21, 2017 at 12:58:06PM -0700, Adam Williamson wrote:
On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=1430297
This has been going on for a few weeks with no attention. It prevents anyone from running libguestfs on Fedora 26.
It's (IMHO) pretty serious for anyone using virtualization: "kernel doesn't boot on qemu" should be a blocker for a Fedora release.
Can anyone see what's going on?
The earlier case looks like https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed in rc2. I'm not sure about the latter case, though Thorsten Leemhuis reported intermittent crash / hang during boot with rc2 as:
https://bugzilla.kernel.org/show_bug.cgi?id=194911
Since rc2 I've seen occasional failures which are probably the same as the ones Thorsten was seeing, but usually it works OK. I run 26-on-26 VMs via virt-manager quite a lot, they certainly don't fail every time.
Does libguestfs-test-tool work for you in F26?
Hum, interesting results:
If I run it as a regular user, it works. If I run it as root as usual, it gives me a libvirt error, some permission issue. If I run it as root with LIBGUESTFS_BACKEND=direct , it hits the same kernel trace you hit.
On Tue, 2017-03-21 at 15:59 -0700, Adam Williamson wrote:
On Tue, 2017-03-21 at 20:48 +0000, Richard W.M. Jones wrote:
On Tue, Mar 21, 2017 at 12:58:06PM -0700, Adam Williamson wrote:
On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=1430297
This has been going on for a few weeks with no attention. It prevents anyone from running libguestfs on Fedora 26.
It's (IMHO) pretty serious for anyone using virtualization: "kernel doesn't boot on qemu" should be a blocker for a Fedora release.
Can anyone see what's going on?
The earlier case looks like https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed in rc2. I'm not sure about the latter case, though Thorsten Leemhuis reported intermittent crash / hang during boot with rc2 as:
https://bugzilla.kernel.org/show_bug.cgi?id=194911
Since rc2 I've seen occasional failures which are probably the same as the ones Thorsten was seeing, but usually it works OK. I run 26-on-26 VMs via virt-manager quite a lot, they certainly don't fail every time.
Does libguestfs-test-tool work for you in F26?
Hum, interesting results:
...
If I run it as root with LIBGUESTFS_BACKEND=direct , it hits the same kernel trace you hit.
Well, first time it did that. Second time, I got this, and the boot hung:
[ 0.937752] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem [ 0.938362] general protection fault: 0000 [#1] SMP [ 0.938719] Modules linked in: kvm_intel kvm snd_pcsp snd_pcm ppdev irqbypass ghash_clmulni_intel snd_timer snd parport_pc ata_generic soundcore joydev pata_acpi parport i2c_piix4 serio_raw qemu_fw_cfg acpi_cpufreq tpm_tis tpm_tis_core tpm libcrc32c crc8 crc7 crc_itu_t virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk virtio_ring virtio nfit crc32_generic crct10dif_pclmul crc32c_intel crc32_pclmul [ 0.939350] CPU: 0 PID: 457 Comm: mount Not tainted 4.11.0-0.rc2.git2.2.fc26.x86_64 #1 [ 0.939350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1.fc26 04/01/2014 [ 0.939350] task: ffff94981d290000 task.stack: ffffae8780374000 [ 0.939350] RIP: 0010:__kmalloc+0xa4/0x210 [ 0.939350] RSP: 0018:ffffae8780377c30 EFLAGS: 00010246 [ 0.939350] RAX: 2d326f6974726976 RBX: 00000000014000c0 RCX: 000000000000000a [ 0.939350] RDX: 0000000000000015 RSI: 0000000000000000 RDI: 000000000001cae0 [ 0.939350] RBP: ffffae8780377c60 R08: ffff94981f01cae0 R09: ffff94981ec02280 [ 0.939350] R10: 00000000000005ac R11: ffff94981b640400 R12: 2d326f6974726976 [ 0.939350] R13: 00000000014000c0 R14: 0000000000002000 R15: ffff94981ec02280 [ 0.939350] FS: 00007fe249b47480(0000) GS:ffff94981f000000(0000) knlGS:0000000000000000 [ 0.939350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.939350] CR2: 000055ac86a6ac08 CR3: 000000001d329000 CR4: 00000000000406f0 [ 0.939350] Call Trace: [ 0.939350] ? mb_cache_create+0x75/0x110 [ 0.939350] mb_cache_create+0x75/0x110 [ 0.939350] ext4_xattr_create_cache+0x13/0x20 [ 0.939350] ext4_fill_super+0x1cad/0x3600 [ 0.939350] ? __kmalloc+0x1d1/0x210 [ 0.939350] ? snprintf+0x49/0x60 [ 0.939350] mount_bdev+0x17c/0x1b0 [ 0.939350] ? ext4_calculate_overhead+0x490/0x490 [ 0.939350] ext4_mount+0x15/0x20 [ 0.939350] mount_fs+0x32/0x160 [ 0.939350] ? __alloc_percpu+0x15/0x20 [ 0.939350] vfs_kern_mount.part.19+0x5d/0x120 [ 0.939350] do_mount+0x520/0xc30 [ 0.939350] ? _copy_from_user+0x4e/0x80 [ 0.939350] ? memdup_user+0x4f/0x80 [ 0.939350] SyS_mount+0x98/0xe0 [ 0.939350] entry_SYSCALL_64_fastpath+0x1a/0xa9 [ 0.939350] RIP: 0033:0x7fe248bcc4fa [ 0.939350] RSP: 002b:00007ffef410b938 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 [ 0.939350] RAX: ffffffffffffffda RBX: 00007fe249741522 RCX: 00007fe248bcc4fa [ 0.939350] RDX: 000055ac86a69c80 RSI: 000055ac86a62f50 RDI: 000055ac86a60240 [ 0.939350] RBP: 00007fe249951184 R08: 0000000000000000 R09: 00007ffef410a838 [ 0.939350] R10: 00000000c0ed0000 R11: 0000000000000246 R12: 000055ac86a60120 [ 0.939350] R13: 00007ffef410bc48 R14: 0000000000000000 R15: 00000000ffffffff [ 0.939350] Code: 49 83 78 10 00 4d 8b 20 0f 84 03 01 00 00 4d 85 e4 0f 84 fa 00 00 00 49 63 41 20 49 8b 39 4c 01 e0 40 f6 c7 0f 0f 85 5f 01 00 00 <48> 8b 18 48 8d 4a 01 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 [ 0.939350] RIP: __kmalloc+0xa4/0x210 RSP: ffffae8780377c30 [ 0.958773] ---[ end trace 900f41ab5fa73f40 ]--- guestfsd: error: /dev/sda1 on / (options: ''): guestfsd: main_loop: proc 1 (mount) took 0.02 seconds libguestfs: error: mount: /dev/sda1 on / (options: ''): libguestfs: closing guestfs handle 0x5653569ae490 (state 2) guestfsd: main_loop: new request, len 0x28 umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime,block_validity,barrier,user_xattr,acl freq=0 passno=0 umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0 umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0 umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=97320k,mode=755 freq=0 passno=0 umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=240912k,nr_inodes=60228,mode=755 freq=0 passno=0 umount-all: /proc/mounts: fsname=/dev/pts dir=/dev/pts type=devpts opts=rw,relatime,mode=600,ptmxmode=000 freq=0 passno=0 [ 1.120063] tsc: Refined TSC clocksource calibration: 3502.622 MHz [ 1.120580] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x327cfdbd6b4, max_idle_ns: 440795243113 ns
Third, fourth and fifth tries I got the virtcons_probe trace again.
On Tue, Mar 21, 2017 at 04:02:19PM -0700, Adam Williamson wrote:
Well, first time it did that. Second time, I got this, and the boot hung:
[ 0.937752] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem [ 0.938362] general protection fault: 0000 [#1] SMP [ 0.938719] Modules linked in: kvm_intel kvm snd_pcsp snd_pcm ppdev irqbypass ghash_clmulni_intel snd_timer snd parport_pc ata_generic soundcore joydev pata_acpi parport i2c_piix4 serio_raw qemu_fw_cfg acpi_cpufreq tpm_tis tpm_tis_core tpm libcrc32c crc8 crc7 crc_itu_t virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk virtio_ring virtio nfit crc32_generic crct10dif_pclmul crc32c_intel crc32_pclmul [ 0.939350] CPU: 0 PID: 457 Comm: mount Not tainted 4.11.0-0.rc2.git2.2.fc26.x86_64 #1 [ 0.939350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1.fc26 04/01/2014 [ 0.939350] task: ffff94981d290000 task.stack: ffffae8780374000 [ 0.939350] RIP: 0010:__kmalloc+0xa4/0x210 [ 0.939350] RSP: 0018:ffffae8780377c30 EFLAGS: 00010246 [ 0.939350] RAX: 2d326f6974726976 RBX: 00000000014000c0 RCX: 000000000000000a [ 0.939350] RDX: 0000000000000015 RSI: 0000000000000000 RDI: 000000000001cae0 [ 0.939350] RBP: ffffae8780377c60 R08: ffff94981f01cae0 R09: ffff94981ec02280 [ 0.939350] R10: 00000000000005ac R11: ffff94981b640400 R12: 2d326f6974726976 [ 0.939350] R13: 00000000014000c0 R14: 0000000000002000 R15: ffff94981ec02280 [ 0.939350] FS: 00007fe249b47480(0000) GS:ffff94981f000000(0000) knlGS:0000000000000000 [ 0.939350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.939350] CR2: 000055ac86a6ac08 CR3: 000000001d329000 CR4: 00000000000406f0 [ 0.939350] Call Trace: [ 0.939350] ? mb_cache_create+0x75/0x110 [ 0.939350] mb_cache_create+0x75/0x110 [ 0.939350] ext4_xattr_create_cache+0x13/0x20 [ 0.939350] ext4_fill_super+0x1cad/0x3600 [ 0.939350] ? __kmalloc+0x1d1/0x210 [ 0.939350] ? snprintf+0x49/0x60 [ 0.939350] mount_bdev+0x17c/0x1b0 [ 0.939350] ? ext4_calculate_overhead+0x490/0x490 [ 0.939350] ext4_mount+0x15/0x20 [ 0.939350] mount_fs+0x32/0x160 [ 0.939350] ? __alloc_percpu+0x15/0x20 [ 0.939350] vfs_kern_mount.part.19+0x5d/0x120 [ 0.939350] do_mount+0x520/0xc30 [ 0.939350] ? _copy_from_user+0x4e/0x80 [ 0.939350] ? memdup_user+0x4f/0x80 [ 0.939350] SyS_mount+0x98/0xe0 [ 0.939350] entry_SYSCALL_64_fastpath+0x1a/0xa9 [ 0.939350] RIP: 0033:0x7fe248bcc4fa [ 0.939350] RSP: 002b:00007ffef410b938 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 [ 0.939350] RAX: ffffffffffffffda RBX: 00007fe249741522 RCX: 00007fe248bcc4fa [ 0.939350] RDX: 000055ac86a69c80 RSI: 000055ac86a62f50 RDI: 000055ac86a60240 [ 0.939350] RBP: 00007fe249951184 R08: 0000000000000000 R09: 00007ffef410a838 [ 0.939350] R10: 00000000c0ed0000 R11: 0000000000000246 R12: 000055ac86a60120 [ 0.939350] R13: 00007ffef410bc48 R14: 0000000000000000 R15: 00000000ffffffff [ 0.939350] Code: 49 83 78 10 00 4d 8b 20 0f 84 03 01 00 00 4d 85 e4 0f 84 fa 00 00 00 49 63 41 20 49 8b 39 4c 01 e0 40 f6 c7 0f 0f 85 5f 01 00 00 <48> 8b 18 48 8d 4a 01 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 [ 0.939350] RIP: __kmalloc+0xa4/0x210 RSP: ffffae8780377c30 [ 0.958773] ---[ end trace 900f41ab5fa73f40 ]---
That's a (another, different) bug. Does it happen often? You said one time in 6 ...
If it's repeatable, then I guess it's another candidate to go into BZ.
Unfortunately filing Fedora kernel BZs is IME not productive. They sit there forever until a "MASS BUG UPDATE" message is added to all kernel bugs, and then they're closed, so really it's a waste of everyone's time. I only bothered to file 1430297 because I needed a bug number to go into my spec file so I could remember why I disabled the tests.
If I have time today I'll try to see if I can bisect the 1430297 issue.
Rich.
On Wed, Mar 22, 2017 at 7:28 AM, Richard W.M. Jones rjones@redhat.com wrote:
On Tue, Mar 21, 2017 at 04:02:19PM -0700, Adam Williamson wrote:
Well, first time it did that. Second time, I got this, and the boot hung:
[ 0.937752] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem [ 0.938362] general protection fault: 0000 [#1] SMP [ 0.938719] Modules linked in: kvm_intel kvm snd_pcsp snd_pcm ppdev irqbypass ghash_clmulni_intel snd_timer snd parport_pc ata_generic soundcore joydev pata_acpi parport i2c_piix4 serio_raw qemu_fw_cfg acpi_cpufreq tpm_tis tpm_tis_core tpm libcrc32c crc8 crc7 crc_itu_t virtio_pci virtio_mmio virtio_input virtio_balloon virtio_scsi nd_pmem nd_btt virtio_net virtio_crypto crypto_engine virtio_console virtio_rng virtio_blk virtio_ring virtio nfit crc32_generic crct10dif_pclmul crc32c_intel crc32_pclmul [ 0.939350] CPU: 0 PID: 457 Comm: mount Not tainted 4.11.0-0.rc2.git2.2.fc26.x86_64 #1 [ 0.939350] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1.fc26 04/01/2014 [ 0.939350] task: ffff94981d290000 task.stack: ffffae8780374000 [ 0.939350] RIP: 0010:__kmalloc+0xa4/0x210 [ 0.939350] RSP: 0018:ffffae8780377c30 EFLAGS: 00010246 [ 0.939350] RAX: 2d326f6974726976 RBX: 00000000014000c0 RCX: 000000000000000a [ 0.939350] RDX: 0000000000000015 RSI: 0000000000000000 RDI: 000000000001cae0 [ 0.939350] RBP: ffffae8780377c60 R08: ffff94981f01cae0 R09: ffff94981ec02280 [ 0.939350] R10: 00000000000005ac R11: ffff94981b640400 R12: 2d326f6974726976 [ 0.939350] R13: 00000000014000c0 R14: 0000000000002000 R15: ffff94981ec02280 [ 0.939350] FS: 00007fe249b47480(0000) GS:ffff94981f000000(0000) knlGS:0000000000000000 [ 0.939350] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.939350] CR2: 000055ac86a6ac08 CR3: 000000001d329000 CR4: 00000000000406f0 [ 0.939350] Call Trace: [ 0.939350] ? mb_cache_create+0x75/0x110 [ 0.939350] mb_cache_create+0x75/0x110 [ 0.939350] ext4_xattr_create_cache+0x13/0x20 [ 0.939350] ext4_fill_super+0x1cad/0x3600 [ 0.939350] ? __kmalloc+0x1d1/0x210 [ 0.939350] ? snprintf+0x49/0x60 [ 0.939350] mount_bdev+0x17c/0x1b0 [ 0.939350] ? ext4_calculate_overhead+0x490/0x490 [ 0.939350] ext4_mount+0x15/0x20 [ 0.939350] mount_fs+0x32/0x160 [ 0.939350] ? __alloc_percpu+0x15/0x20 [ 0.939350] vfs_kern_mount.part.19+0x5d/0x120 [ 0.939350] do_mount+0x520/0xc30 [ 0.939350] ? _copy_from_user+0x4e/0x80 [ 0.939350] ? memdup_user+0x4f/0x80 [ 0.939350] SyS_mount+0x98/0xe0 [ 0.939350] entry_SYSCALL_64_fastpath+0x1a/0xa9 [ 0.939350] RIP: 0033:0x7fe248bcc4fa [ 0.939350] RSP: 002b:00007ffef410b938 EFLAGS: 00000246 ORIG_RAX: 00000000000000a5 [ 0.939350] RAX: ffffffffffffffda RBX: 00007fe249741522 RCX: 00007fe248bcc4fa [ 0.939350] RDX: 000055ac86a69c80 RSI: 000055ac86a62f50 RDI: 000055ac86a60240 [ 0.939350] RBP: 00007fe249951184 R08: 0000000000000000 R09: 00007ffef410a838 [ 0.939350] R10: 00000000c0ed0000 R11: 0000000000000246 R12: 000055ac86a60120 [ 0.939350] R13: 00007ffef410bc48 R14: 0000000000000000 R15: 00000000ffffffff [ 0.939350] Code: 49 83 78 10 00 4d 8b 20 0f 84 03 01 00 00 4d 85 e4 0f 84 fa 00 00 00 49 63 41 20 49 8b 39 4c 01 e0 40 f6 c7 0f 0f 85 5f 01 00 00 <48> 8b 18 48 8d 4a 01 4c 89 e0 65 48 0f c7 0f 0f 94 c0 84 c0 74 [ 0.939350] RIP: __kmalloc+0xa4/0x210 RSP: ffffae8780377c30 [ 0.958773] ---[ end trace 900f41ab5fa73f40 ]---
That's a (another, different) bug. Does it happen often? You said one time in 6 ...
If it's repeatable, then I guess it's another candidate to go into BZ.
Unfortunately filing Fedora kernel BZs is IME not productive. They sit there forever until a "MASS BUG UPDATE" message is added to all kernel bugs, and then they're closed, so really it's a waste of everyone's time. I only bothered to file 1430297 because I needed a bug number to go into my spec file so I could remember why I disabled the tests.
If I have time today I'll try to see if I can bisect the 1430297 issue.
It isn't a waste of time, but there are only 2 of us. We do look at everything that comes in, but there is only so much that 2 people can actually chase down in a week. Right now Laura is trying to work through F26 blockers and I am chasing down the CRDA issue in 4.10 (that will be a blocker for F26 release if it isn't solved).
Justin
On Tue, Mar 21, 2017 at 3:59 PM, Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2017-03-21 at 20:48 +0000, Richard W.M. Jones wrote:
On Tue, Mar 21, 2017 at 12:58:06PM -0700, Adam Williamson wrote:
On Tue, 2017-03-21 at 19:40 +0000, Richard W.M. Jones wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=1430297
This has been going on for a few weeks with no attention. It prevents anyone from running libguestfs on Fedora 26.
It's (IMHO) pretty serious for anyone using virtualization: "kernel doesn't boot on qemu" should be a blocker for a Fedora release.
Can anyone see what's going on?
The earlier case looks like https://bugzilla.redhat.com/show_bug.cgi?id=1430043 , which was fixed in rc2. I'm not sure about the latter case, though Thorsten Leemhuis reported intermittent crash / hang during boot with rc2 as:
https://bugzilla.kernel.org/show_bug.cgi?id=194911
Since rc2 I've seen occasional failures which are probably the same as the ones Thorsten was seeing, but usually it works OK. I run 26-on-26 VMs via virt-manager quite a lot, they certainly don't fail every time.
Does libguestfs-test-tool work for you in F26?
Hum, interesting results:
If I run it as a regular user, it works. If I run it as root as usual, it gives me a libvirt error, some permission issue. If I run it as root with LIBGUESTFS_BACKEND=direct , it hits the same kernel trace you hit.
Off the top of my head, I'd guess that the failing cases are using vhost on the host. I've never used vhost in my life, so I have no clue about it.
--Andy