I'll try to summerize our discussion so far.
(1)
Many of us check / try out betas and RCs according to their respective relevancies. So we
don't completly "fly blind" or rely exclusively on automated testing (to my
relief). For me its virtualisation, PostgreSQL, Web service; for Stephen it may be
Cockpit, Active Diretory. I suppose everybody has some areas specifically interested in.
Can we gather our special interests and systematize it a little bit and make sure that key
areas are not just automatically tested?
(2)
Additionally, we should have a look on our technical specifications and our release
critera (Stephen Gallagher proposed that some times ago) and check automatic testing is OK
and where manual testing makes sense, along with a general revisit as Stephen proposed.
(3)
Regarding ARM SBCs several of us have some experiences with it and own one or the other
device. I suggest we gather our knowledge and create a set of reference devices that we
know basically work with Server (i.e. installable, bootable, network connection, storage,
text terminal / keyboard). IoT provides such a list (in a very short form, a bit more
detailed information would be desirable).
(4)
There seems to be agreement that manual testing is useful / necessary for specific more
complex issues, possibly specific for individual changes in a release. Examples would be
the change to systemd-resolved or the modularization of libvirt, which led to
'surprises'. We should plan as a routine, around the time the first beta is
completed, to go through the list of release changes and check for potential issues and
manual testing requirement (and determine who does what).
QUESTION: Do we want to use our next IRC meeting to start a discussion about our technical
specifications and release criteria? (Feedback greatly appreciated)
For some topics worthwhile discussing see mail from Stephen cf.
https://lists.fedoraproject.org/archives/list/server@lists.fedoraproject....
Am 02.11.2021 um 15:33 schrieb Stephen Gallagher
<sgallagh(a)redhat.com>:
> (3)
> And what about x86_64 installation? Most of us will be in possession of this
hardware. But what about tests? When I started a discussion about testing a year ago, Adam
reassured us, release building and testing is heavily automated these days. No need to
worry too much about testing it manually ourselves. But did anyone of us at least worry a
bit?
I ran some of the validation tests for Fedora Server on the Beta and
RC candidates (in particular, the Cockpit and Active Directory
domain-joining tests).