On Sun, 2022-02-27 at 21:55 +0100, Peter Boy wrote:
Hi Dan,
thanks for all the information. It makes me feel better that we, as Server WG, do not
miss anything expected to contribute.
> Am 27.02.2022 um 17:02 schrieb Dan Čermák <dan.cermak(a)cgc-instruments.com>:
>
> ...
> I am not Adam, but I'll ask anyway: what issues did you encounter?
> openQA is pretty versatile and can test *a lot* of scenarios.
With Fedora 33 it was the switch to systemd-resolved. It introduced a
break in name resolution for the internal libvirt network virbr0. And
after an update, it triggered an unexpected service outage for all
servers that worked with that network. That was pretty bad. And until
a solution was found, it took some time. (You have to use a libvirt
hook via script to make configurations via resolvectl afterwards,
which systemd normally finds itself, but can't find with libvirt,
because libvirt configures the interface directly, independent of
NetworkManager).
With F35, it was the switch to modularized libvirt libraries. We had
tried to get a test day, unfortunately unsuccessfully. So we relied
on the usual test procedures. And then after the release was
published, suddenly we noticed that the usual startup and management
procedures didn't work anymore. And all the servers hosting virtual
machines had a problem. And which server doesn't host VMs nowadays.
Fortunately, this could be fixed very quickly, but the damage was
done.
There was a third problem, probably with F34. This was obviously not
that big, I don't remember at the moment.
And with earlier releases there were issues, too. With F30 or F31
there was a bug with systemd-nspawn and SELinux. This has not been
fixed until today. Fortunately there is a workaround. You have to run
about10 SELinux ACE overwrites, manually after the installation.
Nice. And every time you want to administrate something, you have to
set SELinux to permissive. And remember to enable it again
afterwards.
For a server administrator, these experiences do not inspire much
confidence. SELinux, systemd, network configuration, name
resolution, all these are central and indispensable elements in
Fedora and especially in Fedora Server. And obviously, not even those
elements are reliably under control.
So all of those seem a bit vague and without Bugzilla links I can't
quite understand exactly what the bug was in each case. So let me talk
a bit more generally about what we cover right now.
We cover what's in the release criteria. So, we check that the system
can be installed, and that it boots to a console, and you can log in as
a regular user and root. We check that no system services fail on boot.
We check that you can install updates. We check that Cockpit is set up
by default and that you can connect to it. We test quite a few bits of
Cockpit's feature set. We check that you can set up a postgresql
database and connect to that. We check that you can deploy the system
as a FreeIPA server, and deploy another system as a FreeIPA client of
that server, and check various things about that work, including
enrolling a client via Cockpit and doing some admin tasks in the web
UI. We check the FreeIPA and postgresql tests work on a clean upgrade
from the last two releases. All of these things we test with SELinux
enabled.
There's quite a lot of "SELinux, systemd, network configuration, name
resolution" in all the above. But the thing is, potential
configurations of Fedora are essentially endless, and we can't really
reproduce every one.
One specific thing about openQA is that it does not use libvirt. The
tests run in qemu VMs, but openQA's test runner (os-autoinst) launches
these directly, it does not use libvirt. We use qemu usermode
networking for simple tests, and openvswitch networking for tests where
two or more tests need to communicate with each other.
SELinux denials on things we don't test are definitely a thing that
happens, yeah. What's really useful to avoid them in stable releases is
for folks from this group to test their Server workflows *before* the
release goes stable - say, when Beta comes out - and see what's broken.
Then we can get them fixed before the stable release. I used to run my
own mail, web, FreeIPA and radicale servers, and I upgraded my server
VMs a little before the Final freeze so I could catch problems like
that and report them. But maintaining those servers just got to be too
much of a burden and I quit doing it, so I don't catch those things any
more.
--
Adam Williamson
Fedora QA
IRC: adamw | Twitter: adamw_ha
https://www.happyassassin.net