Hello,
I have added some notes below.
The other thing to settle is what we actually want to report. Matt
provided the following guidelines:
- the current state of the subproject
- future plans
- things the team needs from the rest of the project
- any blockers we can help unblock
- big resource requests?
Current state
-------------
I'd say we have a fairly solid process in place now for release
validation, and everyone's probably more or less familiar with it. We
also have the update testing process nailed down, though I still think
we can improve it in some ways, and we *still* want Bodhi 2.0 :)
We have taskotron running for some package tests and openQA for some
installation validation tests. openQA is I think still running on
openSUSE boxes, but we have some work done on containerization to
potentially move them to Fedora hosts (and I've also been working on
packaging the openQA bits for Fedora).
I think we might still be lacking a bit in terms of Change testing;
there's a personpower issue there, though. We usually wind up only
cherrypicking the most obviously potentially disruptive Changes for
testing, and usually from the perspective of 'does this break the
release'; we rarely manage to find the time to test Changes *in
themselves* and help make sure they work well.
I think that is a good summary.
To provide more background for those not familiar with what has been happening lately, we
used openQA (an openSUSE project) in Fedora 22 cycle to test about 20-25 different
installation test cases, using the tester name 'coconut'. We don't
unfortunately have this system running in public (even though Adam's development
machine was), but the pass results were sent into our wiki, and the failures were examined
manually. I think this was quite a success and it saved us a lot of manual work. It also
detected some issues, even though most of those issues were detected manually as well. But
the main benefit here is, I believe, that we can skip a lot of repetitive manual testing
(because we know it's covered by openQA) and focus on handling with real bugs, do more
exploratory testing, etc.
On the Taskotron front, led mainly by Tim and Martin in the recent days, there has been
some improvements and a lot of bug fixes. They are mostly hidden under the hood, but one
thing I would mention was enabling tasks to store "task artifacts" - basically
any files useful for later review. This is currently not yet exposed in the ResultsDB UI,
but the rest of the code is there.
Another invisible area which has been improved is that we've been working our way
towards disposable test clients. Those will allow us to run destructive checks, and
potentially any third-party checks/tasks in the future. This is by no means finished yet.
Future plans
------------
We have a fairly well-defined roadmap for Taskotron development, I
believe; we could broaden its actual test coverage, but I think we
still have some infrastructure work that probably needs nailing down
before we can focus on that, particularly in terms of the disposable
test client work, and also I believe in terms of results management.
Disposable test clients are now our current priority in Taskotron. We also work on
emitting fedmsg notifications, we need that for integration with Bodhi2. Another
middle-term feature could be, I think, finishing up task artifacts to show up in ResultsDB
and then using that functionality to displaying only relevant parts of test logs to
maintainers (e.g. pidgin maintainer should only see test output relevant for pidgin, and
not also everything else) - the last part is actually already implemented, but the dots
are not connected yet.
I'd definitely like to see us extending openQA coverage in the post-F22
'quiet time'. There are still many validation tests that could
successfully be automated, which gives us much better coverage and
reduces the manual testing burden. I'm certainly intending to help work
on this. We could also consider soliciting wider contributions of
openQA tests; I think the openQA experiment has been fairly successful,
and I could certainly envisage some cases where maintainers might want
to have tests related to their packages in openQA.
I don't really know what Adam's vision was in his last sentence. But in general,
yes, automating more installation/fedup/system basics tests seems to be in the plans and
worthwhile.
Something I've been kicking around with release engineering a bit, and
which will probably come up at Flock, is the possibility of revising
the whole TC/RC approach to release composing. It's getting fairly old,
and is built on some assumptions that are becoming somewhat outdated:
that building and consuming composes are inherently expensive
operations and hence need to be 'rationed' and carefully, manually
handled. In an ideal world, I think it would be nice to take a more
modern, CI-ish approach: we should be doing the whole compose process
daily (not just boot.isos and lives) and running automated tests on the
daily composes (openQA is already capable of this), and improving the
efficiency of the process whereby updates get into composes. The
'special package requests' in TC/RC compose requests have become kinda
a permanent feature, but they were never meant to be. I'd prefer if we
could make the process whereby updates move to stable vastly more
efficient, and perhaps have a slightly different process for handling
builds we want to put in TCs/TCs but which we don't yet want to push
stable (the most common case being the packages that are actually
involved in *doing composes*, so in order to test them, we need to
build composes with them) - something more visible and less clunky than
the current 'list them in TC/RC requests and they go into a manually
maintained bleed repo' approach.
Things we need
--------------
Kinda tied in to the above: Bodhi 2.0, and more efficient releng
processes, are the big two I can think of - faster update pushes and
composes. I think tflink and kparal are in a better position than I to
know about any major resources we need.
I don't say we "need" Bodhi2, but it would definitely help us to get rid of
some very nasty code (pushing comments to Bodhi), probably improving speed and reliability
of our checks considerably (we often crash on Bodhi server errors).
Another thing that would help a lot, I think, are some improvements in the TC/RC
engineering process, for example sending fedmsgs after a TC is complete, or having a
better structure metadata describing this compose (for example, a json structure allowing
us to see what different ISO images are available for Server product, their filepaths and
types - DVD, netinst). We have a lot of black magic for this already, and it mostly works,
but such improvements would simplify the code a lot of make it more reliable, especially
when some changes are introduced.
For Taskotron itself, I don't have a feeling we're blocked on something or
urgently need something done, but from the infrastructure point of view, Tim will
definitely be able to provide much better details.