On 11.01.2016 12:08, Marius Vollmer wrote:
Jan Chaloupka <jchaloup(a)redhat.com> writes:
> cockpit integration tests have proofed to be very useful for
> discovering regression and bugs for kubernetes.
I am happy to hear that!
> [...]
>
> Would it be possible to run kubernetes related cockpit integration
> tests on each new build of kubernetes in rawhide?
We have pretty much given up on running our tests on Fedora Rawhide.
- Testing "branched" seems much more relevant than testing
"rawhide".
- Creating a VM image for rawhide seems to require a lot of luck.
- At the time we didn't have the resource to test evey pull request
against rawhide, and without that, we didn't seem to care enough
about the results to actually do something about it in a timely
manner.
That doesn't mean that you can't test rawhide kubernetes. You would just
test it against fedora-testing or fedora-23 instead.
This might all have changed, but once Fedora 24 is branched, I think
we'll put some effort into testing against that for every PR.
Do you really want to test against Rawhide, or are you happy to go along
with our choice of OSes? (Now with Debian! Come while it's green! :)
> Once I tried to run the tests and ended up with downloading a lot of
> images each over 1GB.
Yes, the copmplete set of tests needs quite some images initially, such
as complete FreeIPA and OpenShift installations. This can be reduced if
you only want to run a subset of the tests, but it's only an initial
setup cost so maybe it is better to just eat it once instead of spending
a few days on trying to save one hour of downloading. :)
> How hard it would be to write down a simple script that would expect
> kubernetes build and distribution? As a result, it would set
> everything up and run required tests (without user intervention). Or
> if possible to use your system (possibly Jenkins?) and periodically
> run integration tests (or with a suitable trigger).
I think with the recent addition of "vm-customize" this should not be
hard, not much harder than running the tests in the first place.
Something like this:
$ export TEST_OS=fedora-23
$ ./testsuite-prepare
$ ./vm-customize -i kubernetes-*.rpm
$ ./check-kubernetes
From there you should be able to quite easily add your own non-Cockpit
tests.
This is better than what I was recommending earlier. But you still need
to remember to run '$ sudo ./vm-prep'.
Stef