CC'ing to cockpit-devel mailing list.
On 24.02.2016 13:57, Daniel Mack wrote:
In Brno, we've briefly talked about using Cockpit as extensive systemd DBus API tester. I wanted to follow up on that idea, and prepared a test framework that runs on the CentOS CI infrastructure. This CI test is not yet linked to our GitHub project, but I'm working on that.
The script provisions a host from a pool of bare metal machines with CentOS 7, checks out systemd from a given PR branch, builds it, installs it, and reboots the machine. Once it's back up, it SSHs back into it and then checks out and builds Cockpit. The bits related to Cockpit are here:
https://github.com/systemd/systemd-centos-ci/blob/master/slave/cockpit.sh
As you can see, I'm using a fork of cockpit here, which has one extra patch to disable 3 tests that failed in my test runs:
https://github.com/systemd/cockpit/commit/750f8702
Ideally, I would like to get rid of that and use the official upstream with a specific tag of course. Without this extra patch, however, a 'make check' based on the 0.95 tag results in the following log:
Probably caused by running 'make check' as root. Can try and fix this, or disable the test in that situation.
Kerberos is certainly outside of the scope of what we want to test, so it might be a good idea to make such tests fail gracefully if the server can't be reached, or make them optional in some other way.
In total, the script currently takes around 4 minutes, 45 seconds to complete. While that's not a whole lot, I wonder whether we can speed it up in some way by only building or running a subset of the cockpit code. After all, all we want to test is the systemd DBus API, everything else can be omitted. Is there a sane way to limit the tests to just this?
The first thing to note is that you're running the Cockpit unit tests. These barely interact with systemd.
The integration tests are where the actual testing of cockpit features, such as the systemd interaction happens.
Running the integration tests are documented here:
https://github.com/cockpit-project/cockpit/blob/master/test/README
To try this out, install the stuff documented at the top of the file. And then you would do something like this. Run as non-root:
$ cd test $ sudo ./vm-prep # only necessary the first time $ ./check-verify # runs the entire test suite
But since you want a custom systemd, and not the entire test suite, you would probably end up moving towards something like this:
$ TEST_OS=fedora-23 $ ./vm-reset $ ./vm-download $TEST_OS $ ./vm-install --quick $ ./vm-customize -i systemd*.rpm # Or similar source of binaries $ ./verify/check-system-info # Just one of the tests
You'll note that this is downloading an image (or in the former case) a few images. Jan Scotka has been running this these tests with custom images ... certainly possible. Update the link it the test/images directory.
Anyway ... you'll find that running these tests is a bit more complex and involved. But this is where the real continuous integration stuff happens.
Other projects are starting to run some of them to test various things, and if there's big issues making them work, we can try and solve the issues. Poke someone in #cockpit if you run into trouble.
Stef
Hi Stef,
On 02/24/2016 04:52 PM, Stef Walter wrote:
The first thing to note is that you're running the Cockpit unit tests. These barely interact with systemd.
The integration tests are where the actual testing of cockpit features, such as the systemd interaction happens.
Running the integration tests are documented here:
https://github.com/cockpit-project/cockpit/blob/master/test/README
To try this out, install the stuff documented at the top of the file. And then you would do something like this. Run as non-root:
$ cd test $ sudo ./vm-prep # only necessary the first time $ ./check-verify # runs the entire test suite
But since you want a custom systemd, and not the entire test suite, you would probably end up moving towards something like this:
$ TEST_OS=fedora-23 $ ./vm-reset $ ./vm-download $TEST_OS $ ./vm-install --quick $ ./vm-customize -i systemd*.rpm # Or similar source of binaries $ ./verify/check-system-info # Just one of the tests
You'll note that this is downloading an image (or in the former case) a few images. Jan Scotka has been running this these tests with custom images ... certainly possible. Update the link it the test/images directory.
Hmm, but the machines I'm running this one are more or less stateless, and the scripts literally download gigabytes of images, which also takes a long time. This would need more thinking if we would want to use that as systemd test.
Anyway ... you'll find that running these tests is a bit more complex and involved. But this is where the real continuous integration stuff happens.
Could you elaborate on why it has to be that complex? I would have thought about installing Cockpit onto the test machine and then run some scripts that remote-control the local web interface via tools such as Cucumber or RTest. Wouldn't something like that suffice and make the test setup a whole lot easier to use?
Thanks, Daniel
On 26.02.2016 13:28, Daniel Mack wrote:
Hi Stef,
On 02/24/2016 04:52 PM, Stef Walter wrote:
The first thing to note is that you're running the Cockpit unit tests. These barely interact with systemd.
The integration tests are where the actual testing of cockpit features, such as the systemd interaction happens.
Running the integration tests are documented here:
https://github.com/cockpit-project/cockpit/blob/master/test/README
To try this out, install the stuff documented at the top of the file. And then you would do something like this. Run as non-root:
$ cd test $ sudo ./vm-prep # only necessary the first time $ ./check-verify # runs the entire test suite
But since you want a custom systemd, and not the entire test suite, you would probably end up moving towards something like this:
$ TEST_OS=fedora-23 $ ./vm-reset $ ./vm-download $TEST_OS $ ./vm-install --quick $ ./vm-customize -i systemd*.rpm # Or similar source of binaries $ ./verify/check-system-info # Just one of the tests
You'll note that this is downloading an image (or in the former case) a few images. Jan Scotka has been running this these tests with custom images ... certainly possible. Update the link it the test/images directory.
Hmm, but the machines I'm running this one are more or less stateless, and the scripts literally download gigabytes of images, which also takes a long time.
Typically the runners that run these things cache the images, and only new ones get downloaded. There's a $TEST_DATA environment variable that htelps with this. On a test runner typically pointed to a volume that's mounted persistently.
nd, only the first example I typed above, above downloads multiple images. The second example just uses one.
But if you can't have a persistent volume, then that's a problem.
Luckily there's no need to use the ones we have available for downloading. With a little scripting, such an image could be synthesized by your test runner script. Take a look at test/guest/fedora-23.setup for the kinds of things that are expected.
[1] https://raw.githubusercontent.com/cockpit-project/cockpit/master/test/guest/...
This would need more thinking if we would want to use that as systemd test.
Anyway ... you'll find that running these tests is a bit more complex and involved. But this is where the real continuous integration stuff happens.
Could you elaborate on why it has to be that complex?
Obviously if there's non-sensical complexity, we want to get that out of the way. I'll do a bit of cleanup to help. But as to "why":
Cockpit is an interface for complex and messy operating systems. It interacts directly with tons different projects and they all break, regress, remove API and on and on.
So the integration tests have to actually integrate and boot real operating systems. It ends up being very close to how users would use Cockpit. In most cases they use the exact same build scripts (like spec files) to build cockpit as ends up being built on that operating system ... and so on.
Without the integration testing that we do, the Cockpit project would be dead ... or have a year long stabilization process before a release (ie: as good as dead). But because the tests boot real operating systems, that users are actually running, every week we can release something that works, and catch regressions before they pile up, etc.
Obviously, I said above, you don't have to use the same operating system images we do. If you want to generate your own test machine image, that's possible.
I would have thought about installing Cockpit onto the test machine and then run some scripts that remote-control the local web interface via tools such as Cucumber or RTest. Wouldn't something like that suffice and make the test setup a whole lot easier to use?
phantomjs is what we use as the remote-control. And you could generate a test machine that works with the given tests. I know that Jan Scotka does. And it goes without saying, that if there's issues that get in the way of rolling your own test machine, then we'd be happy to work with you to change things.
Are you fine generating a qcow2 image? If so you would symlink/place it into the test/images directory, and run a test like this:
$ TEST_OS=my-test-image ./verify/check-system-info
If you want to use a test machine you've started elsewhere, then there's probably a couple things that need to be changed in test/testlib.py (included by files like test/verify/check-system-info) to make that work ... and allow just specifying an address to test against, rather than expecting to start a VM.
Let me know if you want to do this second option, it should be easy to make that change.
Stef
On 02/26/2016 02:29 PM, Stef Walter wrote:
On 26.02.2016 13:28, Daniel Mack wrote:
Hi Stef,
On 02/24/2016 04:52 PM, Stef Walter wrote:
The first thing to note is that you're running the Cockpit unit tests. These barely interact with systemd.
The integration tests are where the actual testing of cockpit features, such as the systemd interaction happens.
Running the integration tests are documented here:
https://github.com/cockpit-project/cockpit/blob/master/test/README
To try this out, install the stuff documented at the top of the file. And then you would do something like this. Run as non-root:
$ cd test $ sudo ./vm-prep # only necessary the first time $ ./check-verify # runs the entire test suite
But since you want a custom systemd, and not the entire test suite, you would probably end up moving towards something like this:
$ TEST_OS=fedora-23 $ ./vm-reset $ ./vm-download $TEST_OS $ ./vm-install --quick $ ./vm-customize -i systemd*.rpm # Or similar source of binaries $ ./verify/check-system-info # Just one of the tests
You'll note that this is downloading an image (or in the former case) a few images. Jan Scotka has been running this these tests with custom images ... certainly possible. Update the link it the test/images directory.
Hmm, but the machines I'm running this one are more or less stateless, and the scripts literally download gigabytes of images, which also takes a long time.
Typically the runners that run these things cache the images, and only new ones get downloaded. There's a $TEST_DATA environment variable that htelps with this. On a test runner typically pointed to a volume that's mounted persistently.
nd, only the first example I typed above, above downloads multiple images. The second example just uses one.
But if you can't have a persistent volume, then that's a problem.
Luckily there's no need to use the ones we have available for downloading. With a little scripting, such an image could be synthesized by your test runner script. Take a look at test/guest/fedora-23.setup for the kinds of things that are expected.
[1] https://raw.githubusercontent.com/cockpit-project/cockpit/master/test/guest/...
This would need more thinking if we would want to use that as systemd test.
Anyway ... you'll find that running these tests is a bit more complex and involved. But this is where the real continuous integration stuff happens.
Could you elaborate on why it has to be that complex?
Obviously if there's non-sensical complexity, we want to get that out of the way. I'll do a bit of cleanup to help. But as to "why":
Cockpit is an interface for complex and messy operating systems. It interacts directly with tons different projects and they all break, regress, remove API and on and on.
So the integration tests have to actually integrate and boot real operating systems. It ends up being very close to how users would use Cockpit. In most cases they use the exact same build scripts (like spec files) to build cockpit as ends up being built on that operating system ... and so on.
Without the integration testing that we do, the Cockpit project would be dead ... or have a year long stabilization process before a release (ie: as good as dead). But because the tests boot real operating systems, that users are actually running, every week we can release something that works, and catch regressions before they pile up, etc.
Obviously, I said above, you don't have to use the same operating system images we do. If you want to generate your own test machine image, that's possible.
I would have thought about installing Cockpit onto the test machine and then run some scripts that remote-control the local web interface via tools such as Cucumber or RTest. Wouldn't something like that suffice and make the test setup a whole lot easier to use?
phantomjs is what we use as the remote-control. And you could generate a test machine that works with the given tests. I know that Jan Scotka does. And it goes without saying, that if there's issues that get in the way of rolling your own test machine, then we'd be happy to work with you to change things.
Hi, yep, I'm doing own images (via virt-scripts) to have precise control on base software and then it is prepared to use inside cockpit infrastructure via vm-create script what use this image. (now I'm using creating own virt builder repository, but Marius said that is not needed to do that and do that directly)
Are you fine generating a qcow2 image? If so you would symlink/place it into the test/images directory, and run a test like this:
$ TEST_OS=my-test-image ./verify/check-system-info
If you want to use a test machine you've started elsewhere, then there's probably a couple things that need to be changed in test/testlib.py (included by files like test/verify/check-system-info) to make that work ... and allow just specifying an address to test against, rather than expecting to start a VM.
Let me know if you want to do this second option, it should be easy to make that change.
Stef _______________________________________________ cockpit-devel mailing list cockpit-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/cockpit-devel@lists.fedorahosted....
On 02/26/2016 02:29 PM, Stef Walter wrote:
On 26.02.2016 13:28, Daniel Mack wrote:
Hmm, but the machines I'm running this one are more or less stateless, and the scripts literally download gigabytes of images, which also takes a long time.
Typically the runners that run these things cache the images, and only new ones get downloaded. There's a $TEST_DATA environment variable that htelps with this. On a test runner typically pointed to a volume that's mounted persistently.
I have to ask whether such persistent storage is available.
Obviously if there's non-sensical complexity, we want to get that out of the way. I'll do a bit of cleanup to help. But as to "why":
Cockpit is an interface for complex and messy operating systems. It interacts directly with tons different projects and they all break, regress, remove API and on and on.
So the integration tests have to actually integrate and boot real operating systems. It ends up being very close to how users would use Cockpit. In most cases they use the exact same build scripts (like spec files) to build cockpit as ends up being built on that operating system ... and so on.
Without the integration testing that we do, the Cockpit project would be dead ... or have a year long stabilization process before a release (ie: as good as dead). But because the tests boot real operating systems, that users are actually running, every week we can release something that works, and catch regressions before they pile up, etc.
Obviously, I said above, you don't have to use the same operating system images we do. If you want to generate your own test machine image, that's possible.
Would you expect CentOS 7 to deviate that much that we couldn't rely on the rest of the system (everything except systemd) to remain stable?
The machine my script provisions is a clean CentOS 7 install which can be rebooted and accessed from a control host or other test machine instances. Would it be feasible to install a new version of system and a stable release of Cockpit on one those, and then run the test suite (which would purely act as remote control for Cockpit) on another? Both machines would have the same version of Cockpit checked out, of course. That would be something I could set up.
phantomjs is what we use as the remote-control. And you could generate a test machine that works with the given tests. I know that Jan Scotka does. And it goes without saying, that if there's issues that get in the way of rolling your own test machine, then we'd be happy to work with you to change things.
That's good to know, thanks for that. But still, the entire qcow images dance sounds like a layer that might not be necessary in the CentOS CI, and if possible, I'd like to skip it entirely for this particular use case.
Thanks, Daniel
On 26.02.2016 15:28, Daniel Mack wrote:
On 02/26/2016 02:29 PM, Stef Walter wrote:
On 26.02.2016 13:28, Daniel Mack wrote:
Hmm, but the machines I'm running this one are more or less stateless, and the scripts literally download gigabytes of images, which also takes a long time.
Typically the runners that run these things cache the images, and only new ones get downloaded. There's a $TEST_DATA environment variable that htelps with this. On a test runner typically pointed to a volume that's mounted persistently.
I have to ask whether such persistent storage is available.
Obviously if there's non-sensical complexity, we want to get that out of the way. I'll do a bit of cleanup to help. But as to "why":
Cockpit is an interface for complex and messy operating systems. It interacts directly with tons different projects and they all break, regress, remove API and on and on.
So the integration tests have to actually integrate and boot real operating systems. It ends up being very close to how users would use Cockpit. In most cases they use the exact same build scripts (like spec files) to build cockpit as ends up being built on that operating system ... and so on.
Without the integration testing that we do, the Cockpit project would be dead ... or have a year long stabilization process before a release (ie: as good as dead). But because the tests boot real operating systems, that users are actually running, every week we can release something that works, and catch regressions before they pile up, etc.
Obviously, I said above, you don't have to use the same operating system images we do. If you want to generate your own test machine image, that's possible.
Would you expect CentOS 7 to deviate that much that we couldn't rely on the rest of the system (everything except systemd) to remain stable?
The machine my script provisions is a clean CentOS 7 install which can be rebooted and accessed from a control host or other test machine instances. Would it be feasible to install a new version of system and a stable release of Cockpit on one those, and then run the test suite (which would purely act as remote control for Cockpit) on another? Both machines would have the same version of Cockpit checked out, of course. That would be something I could set up.
Yup, should be possible. The image will need to have the right SSH keys, and then a few other things need to happen in the test suite (see below).
You'll probably end up select certain tests to run, rather than running a whole test suite. That's since some of the tests do destructive things to the system, or change the state in unexpected ways. But lets get this working, and then you can pick and choose.
But to start needs the following work:
1. Make verify/check-xxxx take an address that they use to test against rather than starting a VM. Some work on that done here:
https://github.com/cockpit-project/cockpit/pull/3848
2. Get the Cockpit testsuite running with CentOS ... talked about this at devconf. Should mostly work, but every operating system wants to be different.
Also here's a bit of work to clean up the test/ directory. Lots of folks have added test suites and other stuff there recently, so time for a spring cleaning.
https://github.com/cockpit-project/cockpit/pull/3846
phantomjs is what we use as the remote-control. And you could generate a test machine that works with the given tests. I know that Jan Scotka does. And it goes without saying, that if there's issues that get in the way of rolling your own test machine, then we'd be happy to work with you to change things.
That's good to know, thanks for that. But still, the entire qcow images dance sounds like a layer that might not be necessary in the CentOS CI, and if possible, I'd like to skip it entirely for this particular use case.
Well try it out. You may find that it's simpler to use the images for tests that want to reset the state between each test, bring up multiple machines etc.... Or you might find those are not relevant tests. Lets see.
Stef
cockpit-devel@lists.fedorahosted.org
-
Daniel Mack -
Jan Scotka -
Stef Walter