On 11/9/18 11:32 AM, Alexander Bokovoy via FreeIPA-devel wrote:
> On pe, 09 marras 2018, Florence Blanc-Renaud via FreeIPA-devel wrote:
>> Hi Developers,
>>
>> Currently our test suite contains 3 different test sets:
>> - gating: executed on each pull request
>> - nightly_master: executed monday, wednesday, friday
>> - nightly_rawhide: not executed
>>
>> Some tests are part of both gating and nightly, but that is an
>> exception. The majority of gating tests are not part of nightly, and
>> this was a choice done earlier in the project.
>>
>> Christian raised a concern with this decision (see
>>
https://github.com/freeipa/freeipa/pull/2540#issuecomment-437078271)
>> and would like nightly to also contain all the gating tests.
>>
>> So what's your opinion on this? Should we keep gating and nightly
>> disjoint or make gating a subset of nightly?
I do think that having all gating tests included in nightly does not
hurt... I vote +1 for this. A different story would be if we would be in
serious lack of resources, but I do not think that's the case.
I think we can try to figure out what is our resources capacity and
based on that decide if we run nightly once per day and mix/alternate
different flavors: f28, f29, and rawhide. Also, having a dashboard to
show the last results/statistics will help.
>> I don't have any strong opinion on this, except that we should be
>> consistent...
> I would have nightly running all tests if we can guarantee that at least
> one nightly run is using up to date Fedora repositories, including
> updates-testing.
>
> With such configuration we would be testing:
> - FreeIPA as in the git repo against a known Fedora configuration
> ("stable Fedora for FreeIPA development")
This is basically our nightly_master.
> - FreeIPA as in the git repo against an up-to-date Fedora configuration
> ("stable FreeIPA for Fedora development")
and IIUC this one is the nightly_rawhide.
>
> The latter one is very important to have to allow us automated upgrade
> of our container images. Right now we fix Fedora container image in time
> and then gate/run nightlies against that image. An update of the image
> is almost always causing a breakage to us and require to find out what's
> wrong, usually month or so after the actual breakage happened in Fedora.
I totally agree with this.
>
> If we would have at least one weekly run that uses an updated container
> image, we can automatically switch to that image as a base for our tests
> in case of a success. We also will have a better way to investigate the
> differences if there are nightly runs with the same code base (git
> committish) against two different container images.
+1
I think if everybody agree, then Eagle Team can start planning these
changes. Taking into account that we are working on migrating to f29, it
would be the right moment to apply these changes.
I'm ok with it as long as we have some way to monitor how long it is
running. We don't want to have to rename it from nightly to weekly
because that's how long it takes :-P
rob