timing-critical tests - depcheck, upgradepath

Kamil Paral kparal at redhat.com
Thu Sep 30 09:50:49 UTC 2010


This ticket:
https://fedorahosted.org/autoqa/ticket/228
and this patch:
https://fedorahosted.org/pipermail/autoqa-devel/2010-September/001195.html
forced me to do some thinking.

We have some tests that are really timing-critical - depcheck and upgradepath.
We can't run them in an arbitrary time as we can do with rpmlint and similar.
We must run them just before the packages are pushed - not earlier, not later.

In other words, we are given set of packages to examine and a repository
in some well-defined state. We examine the packages and the repo and say
"OK" or "not OK". Someone (Bodhi) must guarantee that neither the repo nor
the set of packages change between our examination and actual push of those
packages. There must be no steps in between. Otherwise our result would be
misleading.

For upgradepath we have even stricter requirements - *no* repo must change
between our examination and actual push.

Now that we know the requirements, we can ask some questions:

As for the ticket:
Can we really rely on checking koji -pending tags, instead of communicating
with Bodhi? I don't know how the -pending tags are implemented. But can we
be sure that Bodhi takes exactly same set of packages in the exactly same
order as we do, when it tries to push them?

As for the patch:
I have a feeling that for these aforementioned tests there's not really any
sense in sending email reports. These tests either pass or fail, just before
push. If our test fails, then the push fails and maintainers will be probably
notified by Bodhi anyway, won't they?
And more importantly, we currently run upgradepath when packages are 
*requested* for push, but the actual push might be some hours away, is that
right? So the result we produce is not really valid, in the meantime a lot
of things may change. That also speaks against sending some notifications.

But maybe I just got some stuff wrong. What do you think?


More information about the autoqa-devel mailing list