On Mon, May 19, 2014 at 09:33:49AM +0200, Lukas Slebodnik wrote:
On (17/05/14 01:14), Nikolai Kondrashov wrote:
>On 05/16/2014 11:32 PM, Lukas Slebodnik wrote:
>>On (16/04/14 21:30), Nikolai Kondrashov wrote:
>>>In my case there is no beneficial performance difference when running with
>>>valgrind. There is about 25% improvement using /dev/shm when running with
>>>coverage enabled, but even in a VM it takes less than a minute on my
machine.
>>>
>>I was waiting for pshing patch to the upstream.
>>This is a reason of late reply.
>>
>>>I will use /dev/shm and will mess with moving test results around, when/if
>>>I'll need to tune the performance, but for now I'd prefer to keep my
CI code
>>>simple and use relative directories.
>>>
>>I have totally different results. The most problematic is sysdb test,
>>because there is a lof of IO operations.
>
It might look I was exaggerating in my previous mail.
There are results of execution of all tests.
* tests were precompiled (make tests)
* I executed tests with "make -j4 check", because I have 4-core CPU
ramdisk:
time make -j4 check V=0
// snip
real 0m12.462s
user 0m4.887s
sys 0m0.812s
HDD:
time make -j4 check V=0
// snip
real 9m44.722s
user 0m11.920s
sys 0m8.038s
"ramdisk" is 58x faster then HDD.
3 tests failed on HDD due to timeout.
src/tests/sysdb-tests.c:1399:E:SYSDB Tests:test_sysdb_get_new_id:0: (after this
point) Test timeout expired
src/tests/sysdb_ssh-tests.c:202:E:SYSDB_SSH Tests:store_one_host_test:0: (after
this point) Test timeout expired
src/tests/simple_access-tests.c:122:S:user allow/deny:test_both_empty:0: (after
this point) Test timeout expired
The next time, I executed test withour running tests simultaneously: make check
ramdisk:
time make check V=0
// snip
real 0m25.722s
user 0m4.573s
sys 0m0.785s
HDD:
time make check V=0
// snip
real 11m3.424s
user 0m9.509s
sys 0m7.085s
"ramdisk" is 26x faster then HDD.
>I admit I didn't test this thoroughly and might have hit a corner case and
>perhaps I was biased. I'll retry my tests properly once I get to the point of
>optimizing performance.
Running tests in ramdisk is necessary for developers to pass tests
in reasonable time. It is not optimization, it is well known fact
and I will appreciate if continous integration is aware of this fact.
I wonder if we should raise the CK_TIMEOUT variable (or its equivalent)
in the tests? I'm not sure if requiring the tests to be ran in ramdisk
is the right thing to do, mostly because --with-test-dir only defaults
to shm in our specfile, not the source.
So I think we should either set the --with-test-dir value to ramdisk by
default (we already set it this way for distcheck) or raise the
CK_TIMEOUT.