Randomness in Testing

Devan Goodwin dgoodwin at rm-rf.ca
Thu Aug 26 15:08:50 UTC 2010


On Thu, Aug 26, 2010 at 10:24 AM, Justin Harris <jharris at redhat.com> wrote:
>
>
> On Thu, 26 Aug 2010, bkearney at redhat.com wrote:
>
>> R we seeing errors because of this?
>
> Not at the moment, but I was noticing this in some of our rspecs.  It is
> not an immediate problem but I started to have flashbacks of previous
> projects I have been on where it was next to impossible to get all tests
> in the suite to pass because there were random values generated for all
> kinds of stuff, and different tests would pass/fail on subsequent runs for
> no real reason except that these values were different.  A big value that
> I see is that we can be very explicit about inputs for the test, and
> outputs we expect.  Often times when calculating inputs, you basically
> have to recompute what the expected output is, mirroring what is happening
> in the actual method under test.  This IMO is kind of defeating the
> purpose, because most likely the same person is writing the test that
> wrote the feature and is baking in whatever assumptions they have about
> the implementation into the test as well.
>
> For name generation - what if you get a name collision on a 'register'
> call which results in an error.  A rerun would probably not, and the test
> runs fine.  This IMO is not a good thing, because you get two runs of the same
> test with different outputs.  It is true that a static name can also have
> a collision, but it is consistent and forces us to solve that issue in a
> more comprehensive way.

Static strings are more likely to cause those collisions, and a
redeploy will often wipe the db and your error would disappear, it
seems like the same scenario to me, two runs, different results.

I agree to a point, certainly common sense needs to be used, we
shouldn't use a random integer for the quantity of a sub/pool for
instance. But for object names (consumers/products/owners), I
genuinely cannot think of how it would be a problem aside from a
collision, which is less likely than with statics. I would also
definitely agree we should never use a random value directly related
to something we're testing, like what characters are suitable for an
owner name. :)

Static strings get copied everywhere (which is just bad), and coming
up with a unique owner name every time we create one is kind of
irritating. My pref would be to let the test util methods stub out
random names for products/consumers/owners.

That said, day to day, this issue is a far cry from a big one for me,
so I'm cool with giving everything an explicit name if that's the
route we wanna go.

Cheers,

Devan
-- 
Devan Goodwin <dgoodwin at rm-rf.ca>
http://rm-rf.ca



More information about the candlepin mailing list