On 03/17/2017 06:23 PM, Adam Williamson wrote:
Hi folks!
So Colin and I had a bit of a chat about openQA Atomic testing
'integration', and one of the things that came up was the state of the
'compose check report' emails for Atomic tests.
If you're signed up to test@ or devel@ you may have noticed that a
'compose check report' mail is sent out for every mainline Fedora
compose openQA tests, with info on the numbers of passed and failed
tests and some other stuff.
When we set up openQA to test the Atomic Host OStree installer image
and to test the nightly 'two-week Atomic' composes, I asked whether we
should generate those mails, and who they should go to.
The answer I got - and the way it's configured right now - was that
people only wanted the mails produced when a test *failed*, and they
should be mailed to this list plus to one person directly (I think at
first it was Adam Miller, it now seems to be Mike McGrath).
I've just verified that this is actually working: when openQA tests the
'two-week Atomic' composes and a failure occurs, the mail does get
sent. However, there hasn't been a failure of the test since - AFAICS -
2016-10-01. So that's why no such mails have been sent out recently. I
got Mike to check, and he actually did get a mail on 20167-10-01, when
the test last failed.
woot for passing tests! Adam, do you know if there is any way to send
a weekly report that basically states how many tests ran and how many
passed? That would at least let us know that they were there and still
running and passing.
I imagine now that the tests results are in resultsdb the answer is
going to be something related to that.
I think the mails that have been sent to cloud@ were never approved
through moderation, so they never actually appeared here. It'd be good
if a list moderator could check if they see a few 'compose check' mails
hung up in moderation, or something.
I found this thread from a long time ago on this topic:
https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.o...
Looks like someone tried to add that mail address to the list but
doesn't seem to be working. These emails are not in the held messages
from what I can tell. I think I am a moderator and not an
administrator.
https://wiki.list.org/DOC/Difference%20Between%20a%20Moderator%20and%20Ad...
I need to get jzb to make me an admin so I can investigate.
So, I just wanted to give a quick heads-up on that, and say that if
anyone would like that configuration changed, I can do it easily
enough. We could have a report generated for every compose as for the
'main' composes, but it'd be quite dull I think, because there's only
one test and it almost always passes. We can also change the list of
addresses that receive the mails when they're sent, if it should be
changed.
Indeed. Thanks.
I think the real answer in the long run is to have dashboards that
display test results in a nice way (i'm talking flashy gui web page
stuff). We could have one dashboard that highlighted failures in
atomic host across different composes and then a dashboard that
highlighted failures/passes in a particular or for a particular
ostree. I'm not asking *YOU* for this, but I think that is the answer
long term. The emails would then have a link to the test results as
well as the dashboard where you can see all test results.
Also do let me know if running more tests on the Atomic installer
image
would be desirable. For now we just run a single test - a straight-
through install test on x86_64 BIOS - since that's all I was asked for
initially. We could run the same test on UEFI, if desired, and we could
run some of the post-install tests that are run on other images, and we
could run some of the install variant tests; I just don't know which
ones are relevant / useful for the Atomic installer image.
Yes! we would love a UEFI test and I think it would actually be good
to run the atomic-host-tests against these images assuming openQA is
the right tool for that job. I thought openQA was more for interactive
install testing, so please let me know where I'm wrong. roshi knows
openQA and atomic-host-tests so he might be able to comment here.
Thanks Adam,
Dusty