[SSSD] [PATCH] CI: Remove Clang analyzer

Nikolai Kondrashov Nikolai.Kondrashov at redhat.com
Mon Oct 27 19:36:38 UTC 2014


On 10/26/2014 12:46 AM, Lukas Slebodnik wrote:
> On (22/10/14 18:21), Nikolai Kondrashov wrote:
>> On 10/22/2014 03:22 PM, Lukas Slebodnik wrote:
>>> On (22/10/14 11:55), Nikolai Kondrashov wrote:
>>>> On 10/17/2014 01:11 PM, Nikolai Kondrashov wrote:
>>>>> The attached patch removes Clang analyzer run from CI as it takes a long time
>>>>> and its results are ignored.
>>>>>
>>>>> I initially expected it to be taken into use quickly, but as that is not
>>>>> happening, we'd better remove it to save valuable CI server time.
>>>>>
>>>>> We can always add it back when a solution to checking it is agreed on and
>>>>> implemented.
>>>>
>>>> Jakub, can we get this merged? It will saves a good deal of time on the CI
>>>> servers.
>>>
>>> BTW: Developers could be still interested in html report from clang.
>>> This is the reason why execution on "git push instance" and execution on local
>>> machine are not the same use case. Ticket #2435 would be solution.
>>>
>>> I can imagine even better solution than Michal's proposal with --test="a,b,c"
>>> We could have some templates which would describe order of executed test.
>>> (for example in YAML). Developer could easily modify order of test. He/she
>>> could have his/her own templates
>>>      a) make discheck, mock
>>>      b) just code coverage
>>>      c) just static analysis
>>>      d) all tests.
>>> It would also allow us to easily disable problematic test on server side.
>>> (temporary use non default template)
>>>
>>>
>>> Travis CI use similar approach. (but it does not work with fedora)
>>> Example:
>>>      template(configuration file)
>>>          https://github.com/karelzak/util-linux/blob/master/.travis.yml
>>>      bash function used in template
>>>          https://github.com/karelzak/util-linux/blob/master/.travis-functions.sh
>>
>> They use configuration file, because the actual CI code is outside the repo.
>> Travis CI is a CI server, and the configuration file is an interface to it. It
>> would be nice if we could do the same with Jenkins. In our case
>> Jenkins-interfacing code lives in Jenkins configuration and what we have in
>> the repo is only the tests themselves.
>>
>> Regardless of CI system used, I'd like to keep our CI tests independent of its
>> architecture so we can easily run them locally and change the CI system if
>> needed (frankly, Jenkins is barely suitable). I.e. if we used Travis CI, I
>> would have still kept the .travis.yml as minimal as possible, basically just
>> invoking contrib/ci/run.
>>
>> The above is not an argument, but just a clarification and separation of
>> apples from oranges.
>>
> You didn't get my point. I don't want to use Travis CI.

Well, at least I got *that* :)

> I just like how they have implemented this. It is not important whether test
> are executed by Travis or Jenkins or locally with ci script. The most
> important part is that tests (.travis-functions.sh) are separated from the
> "test plan" (.travis.yml)

Ah, understood now, sorry.

>> The ability to reorder tests might be useful, but it seems to me the actual
> I was more interested in skipping some test rather than reordering test.

Good :)

>> practical benefit would be swamped by the design, coding, maintenance and
>> performance overhead. Consider a situation where several tests require common
>> setup, but then are reordered to have another test requiring conflicting setup
>> in between them. I.e. hard to do in the general case.
>>
> Tests should be independent, they can share some setup and teardown function,
> but it is totally different. Tests should pass in any order.

Sure.

When costly shared setup/teardown is taken into consideration, performance of
arbitrary reordering drops.

Consider this:

     setup_A
         test_A1
         test_A2
     teardown_A
     setup_B
         test_B1
         test_B2
     teardown_B

To interleave tests you would have to do this:

     setup_A
         test_A1
     teardown_A
     setup_B
         test_B1
     teardown_B
     setup_A
         test_A2
     teardown_A
     setup_B
         test_B2
     teardown_B

With arbitrary ordering, increasing number of tests for each fixture and
increasing fixture cost, performance drops sharply. But since you agree that
reordering is not that important, we're in the clear.

>> The ability to select tests for execution (including more than one) is
>> essential for anything that lasts longer than, say, 10 seconds and is involved
>> in code/test cycle. The existing three sets are, perhaps, bearable for the
>> current number and granularity of tests, but it would be nice to have more
>> flexible selection and we'll definitely need it as number/length of our tests
>> grows.
>>
> Totally agree.

Great :)!

>> That said, I would say a special file with test lists would be unnecessary as
>> long as most of us don't need a number of long and highly-tuned, complicated
>> lists of tests to run. With test selection on the command line we can always
>> have shell aliases or functions defined by users in their .bashrc, executing
>> appropriate CI commands as necessary.
>>
>> Regarding enabling the tests which are excluded from official CI for whatever
>> reason, but are still useful to have, such as the Clang analyzer at the
>> moment, we can have a solution for enabling them from the command line in a
>> similar way.
>>
>> FWIW, Epoxy allows hierarchical organization of tests and at the moment would
>> allow selecting tests on the command line as such:
>>
>>     contrib/ci/run /debug/valgrind /coverage/clang
>>
>> Or like this:
>>
>>     contrib/ci/run /debug
>>
>> Or this (Bash extglob syntax):
>>
>>     contrib/ci/run '/@(debug|coverage)/make*'
>>
>> Enabling disabled tests works like this, currently (patterns work as well):
>>
>>     contrib/ci/run --enable=/coverage/clang
>>
>> The above would run all enabled tests, plus /coverage/clang. If you need to
>> run just that one disabled test you'll need to specify it twice (as enabling
>> and selection are separate):
>>
>>     contrib/ci/run --enable=/coverage/clang /coverage/clang
>>
> Currently, we have just small set of tests. Imagine near future :-). We will
> have lots of integration tests. Developer changes something in IPA provider.
> He is sure that test will not break ldap, krb5 or ad provider. For development,
> he wants to use just minimal set of integration test from IPA provider and some
> specific fest for nss responder and ifp responder.
> (Of course, developer will run full test set at the end)
>
> I agree that it is possible to do it with command line arguments and simplify
> with aliases. The question is what is more comfortable for developer.
> IMHO, modification of file with all tests is simpler that modification of
> alias or very long oneliner; adding tests with arguments
>     --enable=...  --enable=... --exclude=...  --enable=...
>
> Travis has yaml for this purpose. Beaker uses xml. We can have json or plain
> text file in worst case. Structured declaration has some advantages.

Alright, I think we can implement something like that once we get to a
considerable number of tests. It'll be easier to figure out the details when
we actually have them.

Nick



More information about the sssd-devel mailing list