On 03/25/2015 11:28 PM, Lukas Slebodnik wrote:
On (25/03/15 12:09), Nikolai Kondrashov wrote:
> I agree that running on servers is more convenient and efficient for
> developers, if enough resources is provided. You basically don't have to care
> about maintaining your own environment and your machine is free to do
> something else.
>
I really appreciate you realized I was right. It took a year and we wasted
lot of time in heated discussion. But at least my prediction was right.
It wasn't good idea to have single script for two purposes.
I'm sorry, but I didn't mean that.
> I don't like that our full test suite takes such a long time
and that
> developers have to wait in the queue.
>
If I understand it correctly we all agree that nobody want to use CI script
on local machine. So we can drop this requirement and it will simplify a lot of
things.
I still think it's very useful to be able to run on local machine. And,
unfortunately, at least for me it will make it harder to maintain and develop
the script without that.
As we are not limited by local use case we can rework script to suit
better
to git-push use case. I briefly checked time of execution and the full test
suite takes approximatelly 20-25 minutes. We can change sequential evaluation
of tests to the parallel one. It was not possible due requirement for
local usage. (BTW there will be still posibility to run it locally
but it will not be so comfrtable)
The proposed sub-tests are:
* make-check-valgrind
* make-distcheck
* mock-build
* code coverage
* integration test (patches still on review)
The each sub-test will be tested on all currently supported platform.
rhel7, rhel6, debian_testing, fedora20, fedora_rawhide
BTW we shoudl test fedora 21 and fedora 22 as well
So we will need to dedicate 5 * 7 + 1 machines
The time of execution will reduce from 25 minutes to the time of slowest test,
which is curently make-distcheck and it takes approximately 4 minutes.
It will be 6.25 times faster, which is really good improvement.
I see that you would like to improve the performance, which I would like to do
as well. However, I think that we can keep the ability to run CI locally *and*
be able to split it into parallel runs on the servers.
I think we can do that by implementing the long-ago requested ability to run
specific tests and invoking the tests you mentioned separately on separate
machines, whereas if you run locally all tests would run by default.
What I'm afraid might be a problem here is the storage limitations of the Brno
lab, resulting in random failures during package upgrades or builds on spawned
VMs. And Petr said that they don't provide any quotas as space is limited.
I'm really not sure why that's still the case.
We can theoretically have the VMs running all the time, and re-create them
periodically from pristine images, benefiting from COW, while keeping the
delta minimal, and also removing VM spawning latency.
All that would need a good deal of scripting, though (in separate repo, of
course).
I'm still to figure out the situation with our OpenStack instances (both
private and local), but it seems both of these is its own can of worms.
What do you think, Lukas?
Nick