On (27/03/15 13:31), Nikolai Kondrashov wrote:
On 03/25/2015 11:28 PM, Lukas Slebodnik wrote:
>On (25/03/15 12:09), Nikolai Kondrashov wrote:
>>I agree that running on servers is more convenient and efficient for
>>developers, if enough resources is provided. You basically don't have to care
>>about maintaining your own environment and your machine is free to do
>>something else.
>>
>I really appreciate you realized I was right. It took a year and we wasted
>lot of time in heated discussion. But at least my prediction was right.
>It wasn't good idea to have single script for two purposes.
I'm sorry, but I didn't mean that.
>>I don't like that our full test suite takes such a long time and that
>>developers have to wait in the queue.
>>
>If I understand it correctly we all agree that nobody want to use CI script
>on local machine. So we can drop this requirement and it will simplify a lot of
>things.
I still think it's very useful to be able to run on local machine. And,
unfortunately, at least for me it will make it harder to maintain and develop
the script without that.
>As we are not limited by local use case we can rework script to suit better
>to git-push use case. I briefly checked time of execution and the full test
>suite takes approximatelly 20-25 minutes. We can change sequential evaluation
>of tests to the parallel one. It was not possible due requirement for
>local usage. (BTW there will be still posibility to run it locally
>but it will not be so comfrtable)
>
>The proposed sub-tests are:
>* make-check-valgrind
>* make-distcheck
>* mock-build
>* code coverage
>* integration test (patches still on review)
>
>The each sub-test will be tested on all currently supported platform.
>rhel7, rhel6, debian_testing, fedora20, fedora_rawhide
>BTW we shoudl test fedora 21 and fedora 22 as well
>
>So we will need to dedicate 5 * 7 + 1 machines
>
>The time of execution will reduce from 25 minutes to the time of slowest test,
>which is curently make-distcheck and it takes approximately 4 minutes.
>
>It will be 6.25 times faster, which is really good improvement.
I see that you would like to improve the performance, which I would like to do
as well. However, I think that we can keep the ability to run CI locally *and*
be able to split it into parallel runs on the servers.
I think we can do that by implementing the long-ago requested ability to run
specific tests and invoking the tests you mentioned separately on separate
machines, whereas if you run locally all tests would run by default.
What I'm afraid might be a problem here is the storage limitations of the Brno
lab, resulting in random failures during package upgrades or builds on spawned
VMs. And Petr said that they don't provide any quotas as space is limited.
I'm really not sure why that's still the case.
We can theoretically have the VMs running all the time, and re-create them
periodically from pristine images, benefiting from COW, while keeping the
delta minimal, and also removing VM spawning latency.
All that would need a good deal of scripting, though (in separate repo, of
course).
I'm still to figure out the situation with our OpenStack instances (both
private and local), but it seems both of these is its own can of worms.
What do you think, Lukas?
I wrote my proposal.
If you have different one which will result into shorter execution time
then we will all like it.
Any patches which significantly speed up process or will make local execution
more attractive for developers are welcomed.
LS