On Fri, 2010-10-08 at 09:56 -0400, Kamil Paral wrote:
I would like to reach a consensus how ABORTED and CRASHED test
status
should be used. My idea is this:
CRASHED should be used for programming errors. Any fault that is caused
by our code should be reported as CRASHED.
ABORTED, otoh, should be used for irrevocable failures of third-parties.
Every time I catch an exception that is not caused by our code, but
for example it is a network failure, external script crash, etc, and
there's no point in continuing the test, it should be reported as ABORTED.
Both statuses mean a failure for us, but we will be easily able to
distinguish our failures and other services' failures.
Together with this proposal I would change our exception catcher this way:
Status is set to CRASHED on any uncaught Exception, with one
exception -- when error.JobComplete is raised and uncaught, we would
not override the status.
That will allow easily end the test immediately (deep in the call stack)
when some external party fails. We just set self.result to ABORTED and
call raise error.JobComplete.
What do you think?
Certainly makes sense. I'm having a hard time thinking about how you'd
instrument the code to distinguish between the two, but I don't think
that's needed for me to give a +1 :)
Can you give some more examples to help distinguish CRASHED and ABORTED
results?
Thanks,
James