----- Original Message -----
From: "Josh Boyer" <jwboyer(a)fedoraproject.org>
To: "Daniel P. Berrange" <berrange(a)redhat.com>, "Development
discussions related to Fedora"
<devel(a)lists.fedoraproject.org>
Sent: Friday, February 21, 2014 4:53:03 PM
Subject: Re: May I file 1000 bugs aka upstream test suite tracking
On Fri, Feb 21, 2014 at 9:50 AM, Daniel P. Berrange <berrange(a)redhat.com>
wrote:
> On Fri, Feb 21, 2014 at 09:38:56AM -0500, Josh Boyer wrote:
>> > That being said, a lot of packages in Fedora are simply that: packaged
>> > upstreams. Many (most?) package maintainers are not developers of that
>> > package and as such are probably not equipped to add tests to their
>> > systems.
>> >
>> > A better case here would be to find a way to identify those packages
>> > whose upstreams have tests that are not being run in %check. That
>> > probably *would* be considered a bug.
>>
>> Unless there's a decree from FESCo or FPC about requiring this, it's
>> going to be up to the maintainer as to whether using %check to run
>> testsuites is required. A lot of testsuites require external network
>> access and won't work when run under koji. Also, it increases build
>> time and can bloat BuildRequires. It is *not* a clear-cut bug.
>>
>> Personally, I don't think %check is a good idea at all.
>
> I think the benefit depends on the level of patching the Fedora maintainer
> is doing. If they are shipping just vanilla upstream tar.gz then they can
> have a moderate level of confidence in the functionality of their package
> without tests, since you can assume upstream ran their test before release.
> Running the test suite would however confirm that the package has not
> been broken due to a change in an new version of an external library/tool
> it depends on, so is a pretty good idea to enable on balance.
>
> If the maintainer is including any non-trivial patches that I think that
> enabling %check should almost be mandatory to ensure they are not causing
> regressions through their patches.
I don't disagree that testsuites should be run. I just disagree they
should be run in %check. It's a poor replacement for things that
should be run via AutoQA or some other test framework after the build.
In java land tests are usually run as part of the %build and if they not succeed the build
fails. Running tests in %check will be harder to do as one would have to patch them to not
run in %build so buildtimes do not grow too much. I also think that purely looking for
tests in %check would not give correct data and packages that run tests as part of the
regular build procedure should be considered (it's better practice IMHO).
Regarding tests being run after the build - this is yet another story as it requires
packagers to work on having tests available in some form (subpackage?) for running after
the build and many test suites are not intended to be run post build so making them run
that way is non trivial amount of work. We (eclipse) work on having tests subpackages
available for later run but this is slow and tedious process as it requires actually
fixing upstream test suites.
Alexander Kurtakov
Red Hat Eclipse team