Randomness in Testing

Jeff Weiss jweiss at redhat.com
Wed Aug 25 18:44:15 UTC 2010



On Wed, 2010-08-25 at 14:08 -0400, Justin Harris wrote:
> All,
>     I have seen some use of random number/name generation used in tests 
> recently and want to discuss it further here.  I am
> going to go out on a limb and give the opinion that this is not a good 
> thing, and sets a
> bad precedent.  I think that the real value of tests is that they are an 
> easily repeatable, consistent assessment of where
> the code is at a certain point in time.  Tests really start to lose their 
> value when two runs of the same test on the same
> code base can result in two different outcomes. I know that for things 
> like name generation this probably won't happen,
> but as I said it is starting down a slippery slope, and IMHO we should 
> avoid it if possible.
> 
> Note that there is nothing wrong with *arbitrary* values, or values 
> determined at random and hard coded into tests - let's
> just be sure that they are the same on subsequent runs of the same test.
> 
> Thoughts?
> 
>   - Justin
> 
> _______________________________________________
> candlepin mailing list
> candlepin at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/candlepin

We use timestamps like System.currentTimeMillis() in our tests - here's
why.  

1) It makes tests re-runnable.  When tests create "stuff" on the
software under test, you often can't rerun the test - because it will
fail since the resource name already exists. 

2) It allows you to figure out when resources were created just based on
the name. Sometimes this is helpful in debugging.

timestamps work better than random because randoms only fulfill #1
above.

Yes, there is a minor repeatability issue with using timestamps (or any
calculated value), but I think the value of re-runnability far outweighs
the loss here.  The whole reason randoms are used in the first place is
to avoid these collisions that will cause false positives in the test
results.

Jeff




More information about the candlepin mailing list