Hi guys, (note: devel, packaging and test lists) previously I've done a little experiment and counted how many packages are likely to have upstream test suites and how many don't: http://atodorov.org/blog/2013/12/24/upstream-test-suite-status-of-fedora-20/
In general around 35% do have test suites, the rest don't.
My goal is to bring down the number of packages which ship without any sort of test suite inside their code base.
The first step is to identify them and track them in Bugzilla.
My question is: **Is everyone, especially package maintainers OK with me filing 1000+ bugs ?**
Last time I did so (around 100 bugs) it got a few people unhappy so better ask this time!
If you are unhappy seeing such many bugs and having your mailbox full with notifications from Bugzilla please reply with a better proposal and why do you consider it better.
Thanks, Alex
On Fri, Feb 21, 2014 at 04:22:42PM +0200, Alexander Todorov wrote:
Hi guys, (note: devel, packaging and test lists) previously I've done a little experiment and counted how many packages are likely to have upstream test suites and how many don't: http://atodorov.org/blog/2013/12/24/upstream-test-suite-status-of-fedora-20/
In general around 35% do have test suites, the rest don't.
[...]
My question is: **Is everyone, especially package maintainers OK with me filing 1000+ bugs ?**
Last time I did so (around 100 bugs) it got a few people unhappy so better ask this time!
Is it correct that you're only going to be filing bugs when upstream tarballs already contain test suites, but they are just not enabled in the Fedora package? If so, it seems reasonable to file a bug encouraging the packager to enable the existing test suite.
My concern is that the packager is not the right person to create a new test suite for a package.
Rich.
На 21.02.2014 16:27, Richard W.M. Jones написа:
Is it correct that you're only going to be filing bugs when upstream tarballs already contain test suites, but they are just not enabled in the Fedora package?
Hi Richard, I meant just the opposite. However I will also do what you suggest but this will result in far less number of bugs (probably around 100).
I want to track which packages *DO NOT* have any tests and later be able to focus on creating them (be it working with volunteers, GSoC participants or whoever is willing to step up to this task).
I don't intend to force package maintainers to write tests if they don't want to (or can't).
-- Alex
On 21/02/14 14:51, Alexander Todorov wrote:
I want to track which packages *DO NOT* have any tests and later be able to focus on creating them (be it working with volunteers, GSoC participants or whoever is willing to step up to this task).
Why would you file a bug in the Fedora bug tracker when the package has no test suite upstream? That makes no sense - if the upstream package has no tests then the bug belongs upstream not in Fedora.
Tom
On Fri, Feb 21, 2014 at 02:53:55PM +0000, Tom Hughes wrote:
On 21/02/14 14:51, Alexander Todorov wrote:
I want to track which packages *DO NOT* have any tests and later be able to focus on creating them (be it working with volunteers, GSoC participants or whoever is willing to step up to this task).
Why would you file a bug in the Fedora bug tracker when the package has no test suite upstream? That makes no sense - if the upstream package has no tests then the bug belongs upstream not in Fedora.
I don't believe that is what he is proposing. I read it as meaning filing bugs where upstream *does* have a test suite, but the RPM specfile has not enabled it in %check.
Regards, Daniel
On 21/02/14 14:57, Daniel P. Berrange wrote:
On Fri, Feb 21, 2014 at 02:53:55PM +0000, Tom Hughes wrote:
On 21/02/14 14:51, Alexander Todorov wrote:
I want to track which packages *DO NOT* have any tests and later be able to focus on creating them (be it working with volunteers, GSoC participants or whoever is willing to step up to this task).
Why would you file a bug in the Fedora bug tracker when the package has no test suite upstream? That makes no sense - if the upstream package has no tests then the bug belongs upstream not in Fedora.
I don't believe that is what he is proposing. I read it as meaning filing bugs where upstream *does* have a test suite, but the RPM specfile has not enabled it in %check.
That was what I thought he really meant originally, and that was what the wiki page suggested, but it doesn't seem to be what he is saying in the message I quoted.
Tom
На 21.02.2014 16:58, Tom Hughes написа:
On 21/02/14 14:57, Daniel P. Berrange wrote:
On Fri, Feb 21, 2014 at 02:53:55PM +0000, Tom Hughes wrote:
On 21/02/14 14:51, Alexander Todorov wrote:
I want to track which packages *DO NOT* have any tests and later be able to focus on creating them (be it working with volunteers, GSoC participants or whoever is willing to step up to this task).
Why would you file a bug in the Fedora bug tracker when the package has no test suite upstream? That makes no sense - if the upstream package has no tests then the bug belongs upstream not in Fedora.
I don't believe that is what he is proposing. I read it as meaning filing bugs where upstream *does* have a test suite, but the RPM specfile has not enabled it in %check.
That was what I thought he really meant originally, and that was what the wiki page suggested, but it doesn't seem to be what he is saying in the message I quoted.
Guys I can do both.
1) Report packages which *have* test suites but they are *not* executed in %check
2) Report packages which *don't* have any test suites at all.
1) is easy but I'm more interested in 2)
-- Alex
On Friday, 21 February 2014 at 16:08, Alexander Todorov wrote: [...]
Guys I can do both.
Report packages which *have* test suites but they are *not* executed in %check
Report packages which *don't* have any test suites at all.
is easy but I'm more interested in 2)
Could you provide the rationale for 2? Why do you want to do it and what does Fedora gain by doing it?
IMHO, 1 is the only one worth having filed as a bugs in our bugzilla. You can track 2 any way you like, but mass bug filing is the wrong way. It's not a bug in the Fedora package that upstream doesn't have a test suite and I'll be sure to close such bugs as WONTFIX.
Personally, I do try to run the test suite in %check for my packages, provided that it doesn't require (much) patching and is easy to do.
Regards, Dominik
На 21.02.2014 17:16, Dominik 'Rathann' Mierzejewski написа:
On Friday, 21 February 2014 at 16:08, Alexander Todorov wrote: [...]
Guys I can do both.
Report packages which *have* test suites but they are *not* executed in %check
Report packages which *don't* have any test suites at all.
is easy but I'm more interested in 2)
Could you provide the rationale for 2? Why do you want to do it and what does Fedora gain by doing it?
I want to work on improving 2). Though Bugzilla is a nice way to track it. Seems like others think otherwise so I'll stick to the wiki.
-- Alex
On Fri, Feb 21, 2014 at 04:16:59PM +0100, Dominik 'Rathann' Mierzejewski wrote:
- Report packages which *don't* have any test suites at all.
Could you provide the rationale for 2? Why do you want to do it and what does Fedora gain by doing it?
Better code quality, less risk of regressions, easier automatic QA.
I think that this is a good initiative, and absolutely fits with something Fedora can and should help with -- but I also agree that bugzilla probably isn't the best place to track it. (It's kind of a shame, though, because a bug tracker would be nicer than a wiki page. We just don't have the right kind of flexiblity in our bugzilla setup to do it.)
На 21.02.2014 16:57, Daniel P. Berrange написа:
Why would you file a bug in the Fedora bug tracker when the package has no test suite upstream? That makes no sense - if the upstream package has no tests then the bug belongs upstream not in Fedora.
I don't believe that is what he is proposing. I read it as meaning filing bugs where upstream *does* have a test suite, but the RPM specfile has not enabled it in %check.
Daniel, I'm proposing quite the opposite. Tom got it right.
-- Alex
On Fri, Feb 21, 2014 at 05:06:03PM +0200, Alexander Todorov wrote:
На 21.02.2014 16:57, Daniel P. Berrange написа:
Why would you file a bug in the Fedora bug tracker when the package has no test suite upstream? That makes no sense - if the upstream package has no tests then the bug belongs upstream not in Fedora.
I don't believe that is what he is proposing. I read it as meaning filing bugs where upstream *does* have a test suite, but the RPM specfile has not enabled it in %check.
Daniel, I'm proposing quite the opposite. Tom got it right.
In that case you should not file this against Fedora bugzilla. Any such requests should go upstream exclusively.
Regards, Daniel
Looks like reporting missing test suites in Bugzilla is not accepted. I guess it's just me who prefers Bugzilla compared to other media.
I *will use the Wiki* for this.
On the topic of tests not executed in %check I *will use Bugzilla* but Alexander Kurtakov brings up another angle - tests executed in %build. This needs more investigation. I will start a separate thread for it.
Thanks for your answers, Alex
На 21.02.2014 16:53, Tom Hughes написа:
Why would you file a bug in the Fedora bug tracker when the package has no test suite upstream? That makes no sense - if the upstream package has no tests then the bug belongs upstream not in Fedora.
Same reason you file kernel bugs in Bugzilla.redhat.com and not bugzilla.kernel.org.
It's easier to use and to automate than me going around every possible bug tracker in the upstream world and filing bugs.
Also easier for volunteers to find and work on them.
-- Alex
----- Original Message -----
From: "Alexander Todorov" atodorov@redhat.com To: "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org, "Development discussions related to Fedora" devel@lists.fedoraproject.org Cc: "For testing and quality assurance of Fedora releases" test@lists.fedoraproject.org Sent: Friday, February 21, 2014 4:51:52 PM Subject: Re: [Fedora-packaging] May I file 1000 bugs aka upstream test suite tracking
На 21.02.2014 16:27, Richard W.M. Jones написа:
Is it correct that you're only going to be filing bugs when upstream tarballs already contain test suites, but they are just not enabled in the Fedora package?
Hi Richard, I meant just the opposite. However I will also do what you suggest but this will result in far less number of bugs (probably around 100).
I want to track which packages *DO NOT* have any tests and later be able to focus on creating them (be it working with volunteers, GSoC participants or whoever is willing to step up to this task).
These are upstream bugs and having them open in Fedora bz would do no good. While this value is valuable it's not Fedora bug and list of packages that are missing test suites in general would better be kept on e.g. wiki page where volunteers can pick one and work with upstream to add test suite. For packages that has test suites but don't compile/run it's another story.
Alexander Kurtakov Red Hat Eclipse team
I don't intend to force package maintainers to write tests if they don't want to (or can't).
-- Alex -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
On Fri, Feb 21, 2014 at 04:22:42PM +0200, Alexander Todorov wrote:
Hi guys, (note: devel, packaging and test lists) previously I've done a little experiment and counted how many packages are likely to have upstream test suites and how many don't: http://atodorov.org/blog/2013/12/24/upstream-test-suite-status-of-fedora-20/
In general around 35% do have test suites, the rest don't.
My goal is to bring down the number of packages which ship without any sort of test suite inside their code base.
The first step is to identify them and track them in Bugzilla.
My question is: **Is everyone, especially package maintainers OK with me filing 1000+ bugs ?**
Last time I did so (around 100 bugs) it got a few people unhappy so better ask this time!
If you have code that can fairly reliably detect whether a test suite exists in the source tar.gz, then I think you would be justified in filing bugs for spec files which have not enabled the test suite.
What I wouldn't do is blindly mass file bugs against every package and ask the maintainer to investigate whether the test suite exists or not. Basically you want to minimize the risk of false bug reports against tests not being enabled, to avoid wasting maintainers time.
Regards, Daniel
На 21.02.2014 16:55, Daniel P. Berrange написа:
If you have code that can fairly reliably detect whether a test suite exists in the source tar.gz, then I think you would be justified in filing bugs for spec files which have not enabled the test suite.
At present I'm aware of 11 different location (directory or file names) which represent tests. Not sure how accurate is this though.
What I wouldn't do is blindly mass file bugs against every package and ask the maintainer to investigate whether the test suite exists or not.
I try to avoid that to the extent possible.
-- Alex
packaging@lists.fedoraproject.org