On Thu, 13 Feb 2014 03:42:03 -0500 (EST) Kamil Paral kparal@redhat.com wrote:

I've noticed that we started to use the word 'check' for naming the scripts performing package-level or system-level checks/tests. I'd like to be consistent in our code and in our documentation, so that our users are not confused. Can somebody (probably Tim) clarify what should we call a 'check' and what should we call a 'test'? Do I understand correctly that 'checks' are anything provided by the users (the scripts), and 'tests/testing' will be used mainly for denoting unit/functional tests?

It's an attempt by me to differentiate between testing and checking but choosing my battles.

Michael Bolton and James Bach have spent far more time on differentiating between checking and testing [1] than I have but as a summary (borrowing some stuff verbatim from their article):

[1] http://www.satisfice.com/blog/archives/856

From their definitions:

- Testing is the process of evaluating a product by learning about it through experimentation, which includes to some degree: questioning, study, modeling, observation and inference.

- Checking is the process of making evaluations by applying algorithmic decision rules to specific observations of a product.

Within the scope of this specific discussion, the two key implications I see are:

- Testing encompasses checking, whereas checking cannot encompass testing.

- Checking is a process that can, in principle be performed by a tool instead of a human, whereas testing can only be supported by tools. Nevertheless, tools can be used for much more than checking.

Using these definitions, anything run in taskotron that reports a PASS/FAIL/OTHER status (ie, everything) would be a check. I'm not saying that checks are bad or anything like that, just that there are limitations to what a check (and an automated check, in particular) can do.

In the context of testing Fedora, I see a value in differentiating between checking and testing because it highlights the value of human testing contributions. Having automated checks for various parts of Fedora is great and I'm all for the attempt to do get them running but I do not believe that the ideas of automating all testing for Fedora or that all tests could/should be automated are worth considering because they are _impossible_ without incredible advances in AI or the singularity.

While automated checking is a useful and important part of a good overall testing strategy, those automated checks are only a portion of said testing strategy. There is a place for sapient, (semi-)exploratory testing in Fedora and there will be a place for it in the foreseeable future.

That being said, I'm not sure there is enough benefit to fight the common usage of the term "unit testing". While they are checks by the definitions I listed above, the cost of re-defining all "unit tests" as "unit checks" would be rather high and I'm not convinced that there's enough benefit there to justify the attempt.

At the moment, I'm working on porting autoqa.test.TestDetail class into Taskotron. When following these directions, I guess it should be called libtaskotron.check.CheckDetail then?

Yes, pretty much anything that was a test in autoqa would be a check in the context of taskotron.

Tim

Using these definitions, anything run in taskotron that reports a PASS/FAIL/OTHER status (ie, everything) would be a check. I'm not saying that checks are bad or anything like that, just that there are limitations to what a check (and an automated check, in particular) can do.

My main motivation is to give more clarity to emails in this mailing list, because the word 'test' is really overloaded with meaning in our field of work. I think that if we start calling taskotron-runnable tasks 'checks', and use 'tests' only for unit tests etc, it will be easier for all of us to understand what we try to say. And of course the end-user (new check developer) can benefit as well. If I understood you correctly, you're fine with this distinction.

...

At the moment, I'm working on porting autoqa.test.TestDetail class into Taskotron. When following these directions, I guess it should be called libtaskotron.check.CheckDetail then?

Yes, pretty much anything that was a test in autoqa would be a check in the context of taskotron.

OK, I'll update my review request.