On 05/11/2010 07:30 PM, Jeff Spaleta wrote:
On Tue, May 11, 2010 at 3:10 PM, Przemek Klosowski
> This probably means at least a rudimentary application testing rig
> and a discipline that identifies and deals with distressed packages.
Does the ongoing work with AutoQA provide the solution you are looking for?
Yes, it looks like the right approach, and reasonable infrastructure,
but it needs a lot of work on tests, and on the policy of integrating
the tests with the packaging and release process.
Currently the tests seem to only address the packaging issues; tallying
the tests from
gives this distribution:
We need a lot of application-specific tests, which at least presently
would have to be hand-rolled. A reasonable future approach might be the
regression testing, i.e. tests generated out of bug reports. In
principle, the bugzilla informal query on how to replicate the bug could
be broken out and formalized to a point where the data might be usable
to a testing harness.
The policy question is pretty complicated and multifaceted:
* errors have to be classified as to severity/impact;
* each failure requires a decision by the package manager on what is the
appropriate action, and who should be responsible (upstream? manager?);
* such decisions should probably be remembered from test run to test run
to mask test noise, but probably should be reviewed periodically
* what to do if the maintainer or upstream are not responsive
It's a large and complicated task; even carefully maintained projects
have test errors, and in fact such projects tend to have more extensive
test suites so they may end up being noisier at least initially.