Hi Dan,
thanks for all the information. It makes me feel better that we, as Server WG, do not miss
anything expected to contribute.
Am 27.02.2022 um 17:02 schrieb Dan Čermák
<dan.cermak(a)cgc-instruments.com>:
...
I am not Adam, but I'll ask anyway: what issues did you encounter?
openQA is pretty versatile and can test *a lot* of scenarios.
With Fedora 33 it was the switch to systemd-resolved. It introduced a break in name
resolution for the internal libvirt network virbr0. And after an update, it triggered an
unexpected service outage for all servers that worked with that network. That was pretty
bad. And until a solution was found, it took some time. (You have to use a libvirt hook
via script to make configurations via resolvectl afterwards, which systemd normally finds
itself, but can't find with libvirt, because libvirt configures the interface
directly, independent of NetworkManager).
With F35, it was the switch to modularized libvirt libraries. We had tried to get a test
day, unfortunately unsuccessfully. So we relied on the usual test procedures. And then
after the release was published, suddenly we noticed that the usual startup and management
procedures didn't work anymore. And all the servers hosting virtual machines had a
problem. And which server doesn't host VMs nowadays.
Fortunately, this could be fixed very quickly, but the damage was done.
There was a third problem, probably with F34. This was obviously not that big, I don't
remember at the moment.
And with earlier releases there were issues, too. With F30 or F31 there was a bug with
systemd-nspawn and SELinux. This has not been fixed until today. Fortunately there is a
workaround. You have to run about10 SELinux ACE overwrites, manually after the
installation. Nice. And every time you want to administrate something, you have to set
SELinux to permissive. And remember to enable it again afterwards.
For a server administrator, these experiences do not inspire much confidence. SELinux,
systemd, network configuration, name resolution, all these are central and indispensable
elements in Fedora and especially in Fedora Server. And obviously, not even those elements
are reliably under control.
So, perhaps we have developed a kind of allergy or paranoia to change that threatens each
time to add another problem and quick&dirty solution to a situation already stuffed
with "Q&Ds“.
But, </complainMode off>, most things work very well in Fedora and especially Fedora
Server. It's the best I've been able to find so far, hence my commitment.
And therefore I’m looking for ways to find such bugs or issues, that have "slipped
through" so far, before the release date (so we can at least add yet another Q&D
to our collection beforehand). I wonder if bugs like the ones described above are
conceptually detectable at all with automated testing like openQA. As a first step we
started work on an update of our technical specification and subsequently our test and
release criteria. Perhaps we can expand or sharpen our tests on this basis.
I have never dealt with this practically before and therefore have a pretty big knowledge
gap and need to catch up.
Best
Peter
P.S. Sorry for the long text.