On pe, 09 marras 2018, Florence Blanc-Renaud via FreeIPA-devel wrote:
Hi Developers,
Currently our test suite contains 3 different test sets:
- gating: executed on each pull request
- nightly_master: executed monday, wednesday, friday
- nightly_rawhide: not executed
Some tests are part of both gating and nightly, but that is an
exception. The majority of gating tests are not part of nightly, and
this was a choice done earlier in the project.
Christian raised a concern with this decision (see
https://github.com/freeipa/freeipa/pull/2540#issuecomment-437078271)
and would like nightly to also contain all the gating tests.
So what's your opinion on this? Should we keep gating and nightly
disjoint or make gating a subset of nightly?
I don't have any strong opinion on this, except that we should be
consistent...
I would have nightly running all tests if we can guarantee that at
least
one nightly run is using up to date Fedora repositories, including
updates-testing.
With such configuration we would be testing:
- FreeIPA as in the git repo against a known Fedora configuration
("stable Fedora for FreeIPA development")
- FreeIPA as in the git repo against an up-to-date Fedora configuration
("stable FreeIPA for Fedora development")
The latter one is very important to have to allow us automated upgrade
of our container images. Right now we fix Fedora container image in time
and then gate/run nightlies against that image. An update of the image
is almost always causing a breakage to us and require to find out what's
wrong, usually month or so after the actual breakage happened in Fedora.
If we would have at least one weekly run that uses an updated container
image, we can automatically switch to that image as a base for our tests
in case of a success. We also will have a better way to investigate the
differences if there are nightly runs with the same code base (git
committish) against two different container images.
--
/ Alexander Bokovoy
Sr. Principal Software Engineer
Security / Identity Management Engineering
Red Hat Limited, Finland