In case some people on this list missed the thread on Fedora-Devel,
there are some rather nice ideas (or not :) for Bodhi below.
Original email:
http://lists.fedoraproject.org/pipermail/devel/2010-March/131723.html
---------- Forwarded message ----------
From: Aaron Faanes <dafrito(a)gmail.com>
Date: Mon, Mar 1, 2010 at 15:22
Subject: Fight bugs, not FESCo
To: devel(a)lists.fedoraproject.org
<snip>
Look at it this way: In a perfect scenario, every user of a package
runs every test the moment it hits updates-testing. That means that a
package only needs to stay in updates-testing for however long the
package tests take. Barely any time at all! We don't live in an ideal
world, though, but I think the formula looks something like this:
Quality of Product = Quality of Testing
Quality of Testing = Test Coverage * Number of Configurations
Configurations are your users, and coverage is depth. Coverage is also
accuracy: If I spend 30 minutes testing every preference even though
you didn't change any preferences, then I just wasted my time on your
package. The goal is to make tests easy and specific for the tester,
and you'll find more testers and better test coverage.
* Increase coverage through Bodhi
As a tester, I love Bodhi. It shows me new things to test. If I have a
list of bugs that were fixed in this build, they're all test-cases. If
I have a changelog that shows features that were added, each change is
a test-case. Unfortunately, if I have neither, then I can only really
do a smoke test. Here's a few concrete ideas to encourage better
coverage through Bodhi:
- Allow test cases to be added to Bodhi. These already exist for the
RCs, they could optionally be written by maintainers too. It doesn't
have to be formal, though. "Test the printing stuff" is sufficient.
- Integrate changelogs where available. Since these show what changed,
testers can use them to be more efficient and thorough with their
tests, increasing coverage.
- Further integrate Bugzilla and Bodhi. If I'm able to see what bugs
are open for a package through Bodhi, I can check if those bugs were
affected by the build I'm testing. I could also see if a bug is
already open for this build without having to search through Bugzilla,
increasing my efficiency.
- Tweak karma. I feel bad leaving negative karma and I shouldn't! Bug
reports are healthy. Also, positive karma is rare and subjective. I
think they should either be plain comments, or a per-user checkbox for
a given test-case.
Test cases don't have to be formal. We want maintainers and testers to
use these features, not hate them. They shouldn't get in the way. But,
if a maintainer wants users to test an area, it should be easy for him
to let users know and focus their feedback towards that area. Empower
maintainers and testers to figure out what works best.
<snip>
-- Aaron Faanes
--
----------
Mathieu Bridon