Hi -
we (David Kutalek, Izidor Matusov & myself) thought of creating
ABRT test suite and broaden and formalize overall testing efforts.
We came out with mind map (both source XML and bitmap attached)
describing what we have and what we'd like to have in the future.
Basic idea we believe in is that testing in some specific field
should fit the underlying environment. So giving an example:
Testing in Fedora should use Fedora's own infrastructure, that
means e.g. running internal test suite as part of Koji build and
AutoQA tests on Fedora-specific regressions and sanity matters.
Let me describe the mind map in detail, branch by branch:
* Fedora
+ Fedora Test Day
- Global public testing by Fedora community, should focus on
both Features (e.g. Retrace Server), infrastructure
compatibility (e.g. reporting to Bugzilla) and desktop
integration.
+ Fedora AutoQA Tests
- By use of Fedora AutoQA infrastructure we should test only
sanity integration and regressions specific to Fedora only
(e.g. bugzilla plugin worked with
mozilla.org Bugzilla but
not with the Red Hat instance).
+ Release Criteria
- Update to Fedora stable release will be pushed only iff
FedoraQA tests passed & upstream test suite passed in Koji.
* RHEL - Not intended to be discussed publicly.
* Internal Test Suite
+ Unit Tests
- Some folks generally think that writing unit tests *after*
the actual code is written is more or less useless because
the test may match broken code. Although I understand possible
issues it's far superb to what we have now. Also considering
ABRT being still more or less rewritten to enhance it's design,
it's quite a good chance we do it the right way.
We'd like to see unit tests in framework of your choice for
both core and plugins.
+ Regression Tests
- Having wide range of regression tests in sources needs some
time, it may be a pain and it may make you feel like you are
coding tests and not developing the useful code instead.
However, in my opinion, it saves you the time you have to
invest in fixing regressions.
Also with regression tests you may slightly less fear
breakage and anger from users. You may also view it as a step
further in providing more stable code; for example code review
was a positive step in that direction.
+ BuildBot
- BuildBot builds ABRT when new commits we the other day, runs
tests, results sends to the mailing list and perhaps even
copies resulting RPMs to fedorapeople repo for testers
consumption.
* Infrastructure
- Just that ABRT should be installed by default in Fedora and
enable in the GUI. Similar tasks for RHEL and RH
infrastructure.
More on the implementation details:
For Fedora, AutoQA should be the choice. Upstream unit tests
- not sure, up to you. Upstream regression and sanity tests -
we'd prefer Beaker/beakerlib, since we are really familiar
with it and is easy to play with.
Let me know, what you think guys about the mind map, your
frameworks of choice, anything.
Thanks,
Michal