Hello,
As we've recently encountered a significant amount of systemd issues on Fedora Rawhide in upstream, I started playing with an idea of a Fedora Rawhide CI for our upstream systemd repository[0]. We already have a bunch of CI's (like CentOS CI[1], Travis CI[2], etc.), however, none of them runs Fedora. We used to have a Fedora Rawhide CI, but it was rather a hack around Copr, which was neither stable nor maintainable.
I know there's already the Fedora CI, but that's only for Fedora packages (from what I know). In this case we'd need to build & test each GitHub PR. I stumbled upon a Fedora Atomic CI upstream page[3], but haven't found any real-world examples anywhere.
I'm not sure if something like this is possible/feasible in current Fedora infrastructure nor who to contact with such request, so any hints into the right direction would be appreciated (thanks to Adam Williamson for pointing me to this list).
Thank you.
Frantisek
[0] https://github.com/systemd/systemd [1] https://ci.centos.org/computer/systemd-ci-slave01/ [2] https://travis-ci.org/systemd/systemd [3] https://fedoraproject.org/wiki/FedoraAtomicCI/upstream
Hi Frantisek,
On Wed, Apr 17, 2019 at 10:19 AM František Šumšal frantisek@sumsal.cz wrote:
Hello,
As we've recently encountered a significant amount of systemd issues on Fedora Rawhide in upstream, I started playing with an idea of a Fedora Rawhide CI for our upstream systemd repository[0]. We already have a bunch of CI's (like CentOS CI[1], Travis CI[2], etc.), however, none of them runs Fedora. We used to have a Fedora Rawhide CI, but it was rather a hack around Copr, which was neither stable nor maintainable.
Ack
I know there's already the Fedora CI, but that's only for Fedora packages (from what I know). In this case we'd need to build & test each GitHub PR. I stumbled upon a Fedora Atomic CI upstream page[3], but haven't found any real-world examples anywhere.
Right, this is tightly connected to testing only Fedora packages now
I'm not sure if something like this is possible/feasible in current Fedora infrastructure nor who to contact with such request, so any hints into the right direction would be appreciated (thanks to Adam Williamson for pointing me to this list).
Does your team has some time to dedicate to this effort? Or are you looking for a solution which you could just use with small modifications?
Best regards, /M
Thank you.
Frantisek
[0] https://github.com/systemd/systemd [1] https://ci.centos.org/computer/systemd-ci-slave01/ [2] https://travis-ci.org/systemd/systemd [3] https://fedoraproject.org/wiki/FedoraAtomicCI/upstream
-- GPG key ID: 0xFB738CE27B634E4B
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
Hello!
On 4/17/19 10:56 AM, Miroslav Vadkerti wrote:
Hi Frantisek,
On Wed, Apr 17, 2019 at 10:19 AM František Šumšal <frantisek@sumsal.cz mailto:frantisek@sumsal.cz> wrote:
Hello, As we've recently encountered a significant amount of systemd issues on Fedora Rawhide in upstream, I started playing with an idea of a Fedora Rawhide CI for our upstream systemd repository[0]. We already have a bunch of CI's (like CentOS CI[1], Travis CI[2], etc.), however, none of them runs Fedora. We used to have a Fedora Rawhide CI, but it was rather a hack around Copr, which was neither stable nor maintainable.
Ack
I know there's already the Fedora CI, but that's only for Fedora packages (from what I know). In this case we'd need to build & test each GitHub PR. I stumbled upon a Fedora Atomic CI upstream page[3], but haven't found any real-world examples anywhere.
Right, this is tightly connected to testing only Fedora packages now
I'm not sure if something like this is possible/feasible in current Fedora infrastructure nor who to contact with such request, so any hints into the right direction would be appreciated (thanks to Adam Williamson for pointing me to this list).
Does your team has some time to dedicate to this effort? Or are you looking for a solution which you could just use with small modifications?
I'm definitely prepared to delve into this deeper. For example, in case of CentOS CI, we got a Jenkins slave and an API documentation - the rest of the CI was our work.
Best regards, /M
Thank you. Frantisek [0] https://github.com/systemd/systemd [1] https://ci.centos.org/computer/systemd-ci-slave01/ [2] https://travis-ci.org/systemd/systemd [3] https://fedoraproject.org/wiki/FedoraAtomicCI/upstream -- GPG key ID: 0xFB738CE27B634E4B _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org <mailto:ci@lists.fedoraproject.org> To unsubscribe send an email to ci-leave@lists.fedoraproject.org <mailto:ci-leave@lists.fedoraproject.org> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
-- Miroslav Vadkerti :: Principal QE :: BaseOS QE / OSCI IRC mvadkert #osci #baseosci #qe :: GPG 0x7B5B2E95 TPB-C 2C215 :: Mobile +420 773 944 252 Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Wed, 2019-04-17 at 11:19 +0200, František Šumšal wrote:
I'm definitely prepared to delve into this deeper. For example, in case of CentOS CI, we got a Jenkins slave and an API documentation - the rest of the CI was our work.
One thing I forgot to mention: CentOS CI actually hosts a Jenkins instance that replaced the old Fedora infra Jenkins instance:
https://jenkins-fedora-infra.apps.ci.centos.org
This can provide Fedora environments. Though it seems it only does stable releases, and I think only x86_64.
On 17. 04. 19 10:18, František Šumšal wrote:
Hello,
As we've recently encountered a significant amount of systemd issues on Fedora Rawhide in upstream, I started playing with an idea of a Fedora Rawhide CI for our upstream systemd repository[0]. We already have a bunch of CI's (like CentOS CI[1], Travis CI[2], etc.), however, none of them runs Fedora. We used to have a Fedora Rawhide CI, but it was rather a hack around Copr, which was neither stable nor maintainable.
Hey! I usually just test on Fedora in Docker on Travis CI. It kinda works, but of course is a bit slower than using Travis'es Ubuntu.
Examples:
https://github.com/fedora-python/taskotron-python-versions/blob/develop/.tra... https://github.com/fedora-python/taskotron-python-versions/blob/develop/Dock...
https://github.com/fedora-python/pyp2rpm/blob/master/.travis.yml https://github.com/fedora-python/pyp2rpm/blob/master/Dockerfile
https://github.com/frenzymadness/fedora-python-tox/blob/master/.travis.yml https://github.com/frenzymadness/fedora-python-tox/blob/master/Dockerfile
Hey!
On 4/17/19 11:14 AM, Miro Hrončok wrote:
On 17. 04. 19 10:18, František Šumšal wrote:
Hello,
As we've recently encountered a significant amount of systemd issues on Fedora Rawhide in upstream, I started playing with an idea of a Fedora Rawhide CI for our upstream systemd repository[0]. We already have a bunch of CI's (like CentOS CI[1], Travis CI[2], etc.), however, none of them runs Fedora. We used to have a Fedora Rawhide CI, but it was rather a hack around Copr, which was neither stable nor maintainable.
Hey! I usually just test on Fedora in Docker on Travis CI. It kinda works, but of course is a bit slower than using Travis'es Ubuntu.
Examples:
https://github.com/fedora-python/taskotron-python-versions/blob/develop/.tra... https://github.com/fedora-python/taskotron-python-versions/blob/develop/Dock...
https://github.com/fedora-python/pyp2rpm/blob/master/.travis.yml https://github.com/fedora-python/pyp2rpm/blob/master/Dockerfile
https://github.com/frenzymadness/fedora-python-tox/blob/master/.travis.yml https://github.com/frenzymadness/fedora-python-tox/blob/master/Dockerfile
We use Docker in Travis CI as well, and it works fine for unit tests. However, for integration tests, which require a full-fledged systemd instance, we need at least a standard VM.
Ahoj Františku,
On Wednesday, April 17, 2019 10:18:27 AM CEST František Šumšal wrote:
We used to have a Fedora Rawhide CI, but it was rather a hack around Copr, which was neither stable nor maintainable.
I'd like to hear about the blockers in using Copr for CI purposes. Of course, we can not provide jenkins-like workers (copr only builds), but there already is Fedora's Jenkins. So
-> your Github could trigger build in copr -> copr would send event on fedmsg (once build is done) -> your fedora jenkins job could react on that event, and do the testing -> jenkins should be able to let your github repo know
Feel free to ping me on irc, Pavel
Hello Pavel,
On 4/17/19 11:23 AM, Pavel Raiskup wrote:
Ahoj Františku,
On Wednesday, April 17, 2019 10:18:27 AM CEST František Šumšal wrote:
We used to have a Fedora Rawhide CI, but it was rather a hack around Copr, which was neither stable nor maintainable.
I'd like to hear about the blockers in using Copr for CI purposes. Of course, we can not provide jenkins-like workers (copr only builds), but there already is Fedora's Jenkins. So
-> your Github could trigger build in copr -> copr would send event on fedmsg (once build is done) -> your fedora jenkins job could react on that event, and do the testing -> jenkins should be able to let your github repo know
I considered using Copr the "proper way", as it provides the GitHub integration (among others), but I wasn't sure if it's not an "abuse" of the build system. I'm not sure about few things:
1) What kind of machines the Copr pool provides? (containers, KVM, emulated VM, baremetal) As we'd like to run integration test as well, we'd need at least KVM machines (baremetal machines would be preferred)
2) Does the Copr infra has enough resources for this? In CentOS CI we have the Jenkins slave limited to 8 executors, i.e. 8 parallel jobs. During the review periods, this number might be exceeded, but the over-the-limit jobs get properly enqueued. Each such job usually takes from 30 to 60 minutes. I'm not sure about the load-balancing techniques in Copr, but I wouldn't want to end up in a situation where third of the Copr jobs are just systemd. (This might be a gross overestimation just to make a point.)
The huge benefit of using Copr is the availability of non-x86_64 architectures. Right now the only systemd CI, which supports alternative archs, is Ubuntu CI, so this would definitely have an additional value apart from using "just" an another distribution.
Feel free to ping me on irc, Pavel
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
On Wednesday, April 17, 2019 12:35:13 PM CEST František Šumšal wrote:
Hello Pavel,
On 4/17/19 11:23 AM, Pavel Raiskup wrote:
Ahoj Františku,
On Wednesday, April 17, 2019 10:18:27 AM CEST František Šumšal wrote:
We used to have a Fedora Rawhide CI, but it was rather a hack around Copr, which was neither stable nor maintainable.
I'd like to hear about the blockers in using Copr for CI purposes. Of course, we can not provide jenkins-like workers (copr only builds), but there already is Fedora's Jenkins. So
-> your Github could trigger build in copr -> copr would send event on fedmsg (once build is done) -> your fedora jenkins job could react on that event, and do the testing -> jenkins should be able to let your github repo know
I considered using Copr the "proper way", as it provides the GitHub integration (among others), but I wasn't sure if it's not an "abuse" of the build system. I'm not sure about few things:
No, CI/CD in Copr != abuse :-)
- What kind of machines the Copr pool provides? (containers, KVM, emulated
VM, baremetal) As we'd like to run integration test as well, we'd need at least KVM machines (baremetal machines would be preferred)
Copr builds in VMs.
- Does the Copr infra has enough resources for this? In CentOS CI we have
the Jenkins slave limited to 8 executors, i.e. 8 parallel jobs.
There's similar quota for max concurrent builds of one user in Copr.
During the review periods, this number might be exceeded, but the over-the-limit jobs get properly enqueued. Each such job usually takes from 30 to 60 minutes.
Copr has a queue as well, if you have large queue - you have to wait (or manually cancel less important builds, etc.).
I'm not sure about the load-balancing techniques in Copr, but I wouldn't want to end up in a situation where third of the Copr jobs are just systemd. (This might be a gross overestimation just to make a point.)
As said above, one user can not waste all resources in Copr (because quotas, unless there's a bug).
Pavel
The huge benefit of using Copr is the availability of non-x86_64 architectures. Right now the only systemd CI, which supports alternative archs, is Ubuntu CI, so this would definitely have an additional value apart from using "just" an another distribution.
Feel free to ping me on irc, Pavel
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org
Hi there!
I'm so happy you started this thread! Since this is something which our team (packit [1]) is planning to get involved in as it aligns nicely with the Fedora gating work which is being done now by the CPE team.
The biggest problem is the one you already stated: where can we run those tests? I'm pretty sure that a lot of the projects would be fine with running the tests in a container, but some may require a full-blown VM as you do.
One of the obvious candidates for this, as already mentioned multiple times here, is CentosCI. But given the fact that the CPE team is stomped with the amount of work, I'm not sure if they'd be able to take such a big one - full upstream CI for as many Fedora packages as possible. Since in the end it's not just systemd: we'd love to have as many upstream projects run their tests in a Fedora environment as possible. That would allow catching bugs during development so that when the upstream project makes a release, they already know it will work in Fedora rawhide. This is the blessed state we would love to reach.
Franto, I'm not sure how much research you've already did, but please let us know if you found an infrastructure which suits your needs.