proposal: deprecate using raise error.TestFail

James Laska jlaska at redhat.com
Wed Sep 8 19:03:21 UTC 2010


On Wed, 2010-09-08 at 07:41 -0400, Kamil Paral wrote:
> Hi,
> 
> currently many tests use this code in its test objects:
> 
>   raise error.TestFail
> -or-
>   raise error.TestWarn
> 
> to end a test that "failed" (from our perspective). It then creates a few
> ugly exceptions in the output, see here, even if everything went just fine
> (from autotest perspective):
> 
> http://autoqa.fedoraproject.org/results/250-autotest/qa03.c.fedoraproject.org/debug/client.DEBUG
> 
> and finally it reports:
> 
> 09/08 12:48:13 INFO |       job:1254| 		FAIL	rpmlint	rpmlint	timestamp=1283950093	localtime=Sep 08 12:48:13	
> 09/08 12:48:13 INFO |       job:1254| 	END FAIL	rpmlint	rpmlint	timestamp=1283950093	localtime=Sep 08 12:48:13	
> 
> (which is also visible in autotest web frontend).
> 
> However, since we now use jskladan's patchset defining AutoQATest parent 
> class (using self.result variable), I no longer see a reason to raise those 
> exceptions. Quite the contrary.
> Those exception don't even match our result states, which we have a much richer
> set (passed, failed, info, aborted, crashed and maybe some other).
> 
> I'd like to propose that we stop using those autotest-internal TestFail 
> and TestWarn exceptions.
> 
> What will change:
> 1. We no longer see red/purple boxes in autotest frontend.

Hah, I was thinking the same thing when inspecting the autotest tko
results viewer.  It's hard to distinguish between a successful test run
that found problems, vs a test case failure.

> 2. The AutoQATest parent class will be adjusted to automatically output 
> the test result after a test has finished.
> 3. Tests will just need to store proper result in self.result variable,
> that's all. The methods may be then ended in a standard manner (return
> or no statement).
> 
> What will be the benefits:
> 1. We will know that if we see a traceback it's a problem. We won't have to
> examine in detail whether this traceback is "good" or "bad".

+1 agreed

> 2. It will allow us to report any crashed test object (throwing an exception)
> to the mailing list. We have already started to do that and it's great:
> https://fedorahosted.org/pipermail/autoqa-results/2010-September/thread.html
> But currently it works only for those tests that haven't filled self.result
> before the crash occurred. We need to catch all exceptions.
> 
> What do you think?

This is a good improvement, thanks for taking the time to propose.

Thanks,
James

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url : https://fedorahosted.org/pipermail/autoqa-devel/attachments/20100908/ab4ffaef/attachment.bin 


More information about the autoqa-devel mailing list