On Fri, 2022-09-02 at 08:37 +0000, Zbigniew Jędrzejewski-Szmek wrote:
> > Now, because I glued openQA to the critpath because it was handy, there
> > are two sets of consequences to a package being in critical path:
> > 1. Tighter Bodhi requirements
> > 2. openQA tests are run, and results gate the update (except Rawhide)
> > So, one of the implicit questions here is, is it OK to keep twinning
> > these two sets of consequences, or should we split them up? Splitting
> > them up kinda implies answer 2) from my original mail: "Keep the
> > current "critical path" concept but define a broader group
> > of "gated packages" somewhere". Because we would then need some new
> > concept that isn't "critical path". As I said, that's more *work* -
> > it'd require us to write new code in several places. Even if we
> > decide it'd be nice to do this, is it nice *enough* to be worth doing
> > that work?
> I'd still vote for keeping a single critpath list and using it as
> "the list of packages that require extra care and testing".
> As you describe, the original meaning of critpath has shifted, but
> it's because the way we do updates and QA has also shifted. Doing
> gating tests for a package seems much more useful than just keeping
> it longer in 'updates-testing' in hope that somebody discovers an
> important regresion in the second week.
Well, there's a caveat there - openQA doesn't test everything. On the
whole we cover quite a lot with the set of tests that gets run on
updates, but there's certainly lots of potential for there to be
important bugs it misses, that a human tester might catch. So I think
there is still a case for the higher karma requirements too.
> So yeah, I don't think it makes sense to do the extra work to split
> the concepts. Also because we have way too many concepts and processes
> in Fedora already.
On the whole, though, I agree with you. I just don't trust my own
opinion because it's obviously biased by what's convenient for me. :D
> > If we don't think it's worth doing that work, then we're kinda stuck
> > with openQA glomming onto the critpath definition to decide which
> > updates to test and gate, because I don't have any other current viable
> > choices for that, really. And we'd have to figure out a critpath
> > definition that's as viable as possible for both purposes.
> > BTW, one other thought I've had in relation to all this is that we
> > could enhance the current critpath definition somewhat. Right now, it's
> > built out of package groups in comps which are kinda topic-separated:
> > there's a critpath-kde, a critpath-gnome, a critpath-server, and so on.
> > But the generated critical path package list is a monolith: it doesn't
> > distinguish between a package that's on the GNOME critpath and a
> > package that's on the KDE critpath, you just get a big list of all
> > critpath packages. It might be nice if we actually did distinguish
> > between those - the critpath definition could keep track of which
> > critpath topic(s) a package is included in, and Bodhi could display
> > that information in the web UI and provide it via the API. That way
> > manual testers could get a bit more info on why a package is critpath
> > and what areas to test, and openQA could potentially target its test
> > runs to conserve resources a bit, though this might require a bit more
> > coding work on the gating stuff now I think about it.
> That sounds useful. We only need a volunteer to figure out the details
> and do the work ;)
I actually did a huge rewrite of the thing that generates the critpath
data this week, and it probably wouldn't be tooooo much work, honestly.
The most annoying bit would be the Bodhi frontend stuff, but that's
because I'm bad at frontend dev in general. :P But yeah, this is
definitely off in sky-castle land. I'll add it to my ever-growing list
of sky-castle projects to do when I get a couple of years of spare
IRC: adamw | Twitter: adamw_ha
We've had openQA testing of updates for stable and branched releases,
and gating based on those tests, enabled for a while now. I believe
this is going quite well, and I think we addressed the issues reported
when we first enabled gating - Bodhi's gating status updates work more
smoothly now, and openQA respects Bodhi's "re-run tests" button so
failed tests can be re-triggered.
A few weeks ago, I enabled testing of Rawhide updates in the openQA
lab/stg instance. This was to see how smoothly the tests run, how often
we run into unexpected failures or problems, and whether the hardware
resources we have are sufficient for the extra load.
So far this has been going more smoothly than I anticipated, if
anything. The workers seem to keep up with the test load, even though
one out of three worker systems for the stg instance is currently out
of commission (we're using it to investigate a bug). We do get
occasional failures which seem to be related to Rawhide kernel slowness
(e.g. operations timing out that usually don't otherwise time out), but
on the whole, the level of false failures is (I would say) acceptably
low, enough that my current regime of checking the test results daily
and restarting failed ones that don't seem to indicate a real bug
should be sufficient.
So, I'd like to propose that we enable Rawhide update testing on the
production openQA instance also. This would cause results to appear on
the Automated Tests tab in Bodhi, but they would be only informational
(and unless the update was gated by a CI test, or somehow otherwise
configured not to be pushed automatically, updates would continue to be
pushed 'stable' almost immediately on creation, regardless of the
More significantly, I'd also propose that we turn on gating on openQA
results for Rawhide updates. This would mean Rawhide updates would be
held from going 'stable' (and included in the next compose) until the
gating openQA tests had run and passed. We may want to do this a bit
after turning on the tests; perhaps Fedora 37 branch point would be a
natural time to do it.
Currently this would usually mean a wait from update submission to
'stable push' (which really means that the build goes into the
buildroot, and will go into the next Rawhide compose when it happens)
of somewhere between 45 minutes and a couple of hours. It would also
mean that if Rawhide updates for inter-dependent packages are not
correctly grouped, the dependent update(s) will fail testing and be
gated until the update they depend on has passed testing and been
pushed. The tests for the dependent update(s) would then need to be re-
run, either by someone hitting the button in Bodhi or an openQA admin
noticing and restarting them, before the dependent update(s) could be
In the worst case, if updated packages A and B both need the other to
work correctly but the updates are submitted separately, both updates
may fail tests and be blocked. This could only be resolved by waiving
the failures, or replacing the separate updates with an update
containing both packages.
All of those considerations are already true for stable and branched
releases, but people are probably more used to grouping updates for
stable and branched than doing it for Rawhide, and the typical flow of
going from a build to an update provides more opportunity to create
grouped updates for branched/stable. For Rawhide the easiest way to do
it if you need to do it is to do the builds in a side tag and use
Bodhi's ability to create updates from a side tag.
As with branched/stable, only critical path updates would have the
tests run and be gated on the results. Non-critpath updates would be
unaffected. (There's a small allowlist of non-critpath packages for
which the tests are also run, but they are not currently gated on the
I think doing this could really help us keep Rawhide solid and avoid
introducing major compose-breaking bugs, at minimal cost. But it's a
significant change and I wanted to see what folks think. In particular,
if you find the existing gating of updates for stable/branched releases
to cause problems in any way, I'd love to hear about it.
IRC: adamw | Twitter: adamw_ha
Fedora Linux 37 Final freeze begins Tuesday 4 October.
1. anaconda — Custom partitioning with 2 drives selected, bootloader
fails to install due to one drive not having a BIOS Boot partition —
ACTION: QA to verify FEDORA-2022-c20d42f3bc
2. gnome-contacts — When a single contact is edited, it results in
multiple contacts of the same name. — ON_QA
ACTION: QA to verify FEDORA-2022-acbfee2ce8
3. gnome-initial-setup — Unable to set up enterprise account with
gnome-initial-setup, clicking continue does not join the domain —
ACTION: dgilmore to test
4. gnome-shell — GNOME Initial Setup uses the English keyboard,
instead of the default keyboard — ASSIGNED
ACTION: Upstream to diagnose and resolve issue
5. gnome-shell — Attempting to disconnect VPN connection from system
menu does nothing — VERIFIED
6. gnome-software — Can't install a local rpm package anymore, Install
button missing (for certain RPMs) — VERIFIED
7. greenboot — greenboot-grub2-set-counter.service fails on 38 IoT
with "cannot open `/boot/grub2/grubenv.new`: No such file or
directory." — NEW
ACTION: Maintainers to diagnose issue
8. grub2 — Windows with bitlocker enabled can't be booted, needs to
use bootnext instead of chainloader — NEW
ACTION: Maintainers to diagnose issue or provide status update
9. mesa — totem: nouveau_pushbuf_data(): totem killed by SIGABRT — NEW
ACTION: Maintainers to update mesa with fixes for F37
10. uboot-tools — Regression booting Fedora on rockchip devices
installed on PCIe NVME drives — ON_QA
ACTION: QA to verify FEDORA-2022-c1d9e8daa9
1. chrome-gnome-shell — Project name and source repository changed to
gnome-browser-connector — NEW
ACTION: Reviewer to approve new gnome-browser-connector package.
2. gnome-shell — screencast doesn't record the top layer — NEW
ACTION: QA to verify if behavior still exists
3. gnome-shell-extension-background-logo — Update
gnome-shell-extension-background-logo to 43.0 — NEW
ACTION: Maintainer to update package
4. systemd — Czech qwerty layout configured in anaconda, but qwertz
layout used in disk unlock and in VT console — NEW
ACTION: kbd maintainers to reconsider the -legacy subpackage split
1. anaconda — https://bugzilla.redhat.com/show_bug.cgi?id=2088113 — ON_QA
Custom partitioning with 2 drives selected, bootloader fails to
install due to one drive not having a BIOS Boot partition
When using custom partitioning on two drives with /boot on a RAID 1
device, the installer only creates one BIOS boot. grub2-install is run
against both drives, but the second fails because it has no BIOS boot
partition. FEDORA-2022-c20d42f3bc contains a candidate fix.
2. gnome-contacts — https://bugzilla.redhat.com/show_bug.cgi?id=2111003 — ON_QA
When a single contact is edited, it results in multiple contacts of
the same name.
Editing a contact results in the contact being properly updated, but
an empty contact is created with the same name. FEDORA-2022-acbfee2ce8
contains a candidate fix.
3. gnome-initial-setup —
https://bugzilla.redhat.com/show_bug.cgi?id=2123494 — ASSIGNED
Unable to set up enterprise account with gnome-initial-setup, clicking
continue does not join the domain
Initially the buttons were missing. Update FEDORA-2022-50e585b456
fixed that issue, however clicking the button does not joing the
domain. Instead, the user is returned to the domain credentials
screen, resulting in an endless loop.
a backport of upstream's change which may fix it.
4. gnome-shell — https://bugzilla.redhat.com/show_bug.cgi?id=2121110 — ASSIGNED
GNOME Initial Setup uses the English keyboard, instead of the default keyboard
Regardless of the default keyboard setting, g-i-s uses the English
keyboard. Relatedly, new users have the English keyboard selected,
even though another language is shown as the default.
FEDORA-2022-52cdfc7920 failed to fix this issue.
5. gnome-shell — https://bugzilla.redhat.com/show_bug.cgi?id=2125252 — VERIFIED
Attempting to disconnect VPN connection from system menu does nothing
Disabling a VPN doesn't disable the VPN. Fixed in FEDORA-2022-50e585b456.
6. gnome-software —
https://bugzilla.redhat.com/show_bug.cgi?id=2124869 — VERIFIED
Can't install a local rpm package anymore, Install button missing (for
What it says on the tin. Fixed in FEDORA-2022-b6246d02fa.
7. greenboot — https://bugzilla.redhat.com/show_bug.cgi?id=2121944 — NEW
greenboot-grub2-set-counter.service fails on 38 IoT with "cannot open
`/boot/grub2/grubenv.new`: No such file or directory."
greenboot service fails on a file-not-found error. This also results
in rebase tests failing because of conflicts with the rebase triggered
by the greenboot failure.
8. grub2 — https://bugzilla.redhat.com/show_bug.cgi?id=2049849 — NEW
Windows with bitlocker enabled can't be booted, needs to use bootnext
instead of chainloader
Dual-booting recent Windows 10 system with TPM 2.0 fails because
bitlocker can't be unsealed by the TPM. This was waived to F37 under
the "difficult to fix" exception. No additional updates have been
9. mesa — https://bugzilla.redhat.com/show_bug.cgi?id=2123274 — NEW
totem: nouveau_pushbuf_data(): totem killed by SIGABRT
totem plays ~1 second of video and then crashes. This appears to
happen with all video formats. This appears to be an issue in
multithreading support in the nouveau driver, which is fixed upstream.
The fix is large and may not be suitable for backporting, but would be
available in a mesa-22.3 update.
10. uboot-tools — https://bugzilla.redhat.com/show_bug.cgi?id=2124127 — ON_QA
Regression booting Fedora on rockchip devices installed on PCIe NVME drives
uboot-tools introducted a regression when used with 5.19.x kernels.
FEDORA-2022-c1d9e8daa9 contains a candidate fix.
1. chrome-gnome-shell —
https://bugzilla.redhat.com/show_bug.cgi?id=2106868 — NEW
Project name and source repository changed to gnome-browser-connector
Upstream has split this into two packages. Package review pending for
2. gnome-shell — https://bugzilla.redhat.com/show_bug.cgi?id=2125439 — NEW
screencast doesn't record the top layer
Typing and popup windows don't show when recording a screencast. This
may be fixed in pipewire-gstreamer-0.3.57-1.
3. gnome-shell-extension-background-logo —
https://bugzilla.redhat.com/show_bug.cgi?id=2127192 — NEW
Update gnome-shell-extension-background-logo to 43.0
The Fedora logo is not displayed and the package is not compatible
with gnome-shell 43.
4. systemd — https://bugzilla.redhat.com/show_bug.cgi?id=2121106 — NEW
Czech qwerty layout configured in anaconda, but qwertz layout used in
disk unlock and in VT console
The wrong keymap is used when unlocking disks or virtual terminals. It
looks like this may have never worked. It looks like an issue with how
layouts are mapped between console and xkb. Adam is thinking about how
to approach this:
He / Him / His
Fedora Program Manager
F37 CoreOS Test week starts today and will run for a week!
As most of you might know, CoreOS will come forth in this cycle as an "Edition".
This opens us with more requirements for testing and ironing out bugs
well ahead of time.
This test week aims to serve community members an opportunity to get
together and explore
everything from the FCOS itself to the documentation!
The wiki has the details, all that's required to be known. The test
day app is where the testers can submit the test results.
TRIED AND PERSONALLY TESTED, ERGO TRUSTED
For background here, see:
right now, when installing Fedora alongside a Windows install with
BitLocker enabled, trying to boot Windows from the Fedora boot menu
does not work.
We waived the bug as a blocker for Fedora 36 on the basis upstream did
not consider it fixable within the F36 timeframe. We agreed that if
upstream still couldn't get this fixed for F37, we'd consider revising
Well, we're approaching F37 Final and the bug is still open, and
there's no appreciable movement upstream, so I'm proposing the criteria
change. I propose we change this:
"The installer must be able to install into free space alongside an
existing clean Windows installation and install a bootloader which can
boot into both Windows and Fedora."
"The installer must be able to install into free space alongside an
existing clean Windows installation. As long as the Windows
installation does not have BitLocker enabled, the installer must also
install a bootloader which can boot into both Windows and Fedora."
IRC: adamw | Twitter: adamw_ha
I have a clean install of Fedora 37 beta on a desktop (not a VM). When I try to install kodi (from rpmfusion-free) I encounter a conflict with firewalld:
Error: Transaction test error:
file /usr/lib/firewalld/services/kodi-eventserver.xml from install of kodi-firewalld-19.4-4.fc37.x86_64 conflicts with file from package firewalld-1.2.0-3.fc37.noarch
This was reported a few months ago on Arch linux and a bug was filed against firewalld as both were trying to provide the same file. According to github, the firewalld update was reverted back in July: https://github.com/firewalld/firewalld/pull/1002 Apparently that has not made it into Fedora 37 yet?
I have all updates installed as of 23 September.
I'd appreciate any advice.
#fedora-meeting: Fedora QA meeting
Meeting started by adamw at 15:01:54 UTC. The full logs are available at
* Roll Call (adamw, 15:02:01)
* Previous meeting follow-up (adamw, 15:07:41)
* "adamw to work on F37 Beta common issues" - did that, I think
everything nominated was covered. (adamw, 15:08:37)
* Fedora 37 status (adamw, 15:11:28)
* LINK: https://gitlab.gnome.org/GNOME/gnome-maps/-/issues/491
* Release criteria proposal status (adamw, 15:19:42)
* Windows Bitlocker criterion amendment proposal is live, currently
doesn't seem to have any opposition (adamw, 15:35:02)
* ACTION: adamw to activate the Bitlocker amendment this week if
nobody pops up to oppose it (adamw, 15:35:15)
* the GNOME Shell extension criterion proposal was not supported by
the desktop team, so I'm considering that dropped for now (adamw,
* Test Day / community event status (adamw, 15:41:16)
* LINK: https://testdays.fedoraproject.org/events/142 (coremodule,
* IoT Test Day went off last week:
https://testdays.fedoraproject.org/events/142 (adamw, 15:42:52)
* test day publicity was lacking lately as sumantro was out sick and
we missed arranging cover, apologies for that (adamw, 15:51:16)
* CoreOS and upgrade test days are coming soon (adamw, 15:53:42)
* Open floor (adamw, 15:53:47)
Meeting ended at 15:57:42 UTC.
* adamw to activate the Bitlocker amendment this week if nobody pops up
to oppose it
Action Items, by person
* adamw to activate the Bitlocker amendment this week if nobody pops
up to oppose it
People Present (lines said)
* adamw (41)
* zodbot (16)
* tablepc_ (16)
* bcotton (7)
* lruzicka (7)
* coremodule (6)
* SumantroMukherje (5)
* kparal (4)
* Southern_Gentlem (2)
* pboy (1)
Generated by `MeetBot`_ 0.4
.. _`MeetBot`: https://fedoraproject.org/wiki/Zodbot#Meeting_Functions
IRC: adamw | Twitter: adamw_ha
Completely randomly, my laptop sometimes emits a loud PC speaker beep while
rebooting/powering off Fedora 37. The beep is very strong, and as a PC
speaker sound, it of course ignores any configured volume level, mute
status, and even headphones plugged in. I fortunately am in a different
room than the rest of my family sleeps in, but if I were in the same room,
and were I just a regular user, this would probably be the last day of
Fedora on that laptop. The beep is that loud and uncomfortable, especially
I wonder if somebody else running F37 noticed it as well? Any hints what
might cause it and how we can fix it? It never happened on F36 on the same