Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
1) Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
2) One-time enablement of all existing packages
That should be doable. Right?
3) Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
Can we do this? How can I help?
[1] https://fedoraproject.org/wiki/Koschei
On Thu, Aug 25, 2022 at 9:58 AM Miro Hrončok mhroncok@redhat.com wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
- Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
Doesn't koschei have some built-in rate-limiting? If necessary, we could probably just adjust the priority threshold above which scratch builds are triggered.
- One-time enablement of all existing packages
That should be doable. Right?
Yeah that should be pretty simple. Get list of source packages in $release, paste it into the "Add packages" page, set release combobox to $release, submit. I do that regularly for all Rust packages.
- Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
I think so. koschei already knows about all packages - I think it syncs with PDC. So that process could possibly be adapted to not only add the package to the koschei database, but also set the "tracked" boolean to "true" by default instead of "false" by default.
Can we do this? How can I help?
I think we can, and I think we should. Even though scratch builds only run on x86_64, ppc64le, and aarch64 (and don't expose s390x and i686 build failures), it's still a very valuable service. I use it extensively to monitor the Rust stack for new build failures (because I'm pushing dozens of updates every week and I want to know about unintended breakage ASAP).
If there's something that I can do to help make "koschei tracking for all packages on all branches by default" happen, I'll help too :)
Fabio
On Thu, Aug 25, 2022 15:58:12 +0200, Fabio Valentini wrote:
Yeah that should be pretty simple. Get list of source packages in $release, paste it into the "Add packages" page, set release combobox to $release, submit. I do that regularly for all Rust packages.
The rust-sig packages are also set to automatically be tracked by Koschei from the looks of it and I expect this can then be done for all packages instead of for individual SIGs:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/kosc...
On Thu, Aug 25, 2022 at 4:34 PM Ankur Sinha sanjay.ankur@gmail.com wrote:
On Thu, Aug 25, 2022 15:58:12 +0200, Fabio Valentini wrote:
Yeah that should be pretty simple. Get list of source packages in $release, paste it into the "Add packages" page, set release combobox to $release, submit. I do that regularly for all Rust packages.
The rust-sig packages are also set to automatically be tracked by Koschei from the looks of it and I expect this can then be done for all packages instead of for individual SIGs:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/kosc...
As far as I understand, this only automatically adds packages that are associated with the "@rust-sig" group on dist-git to the "rust-sig" group on koschei. We still need to manually enable tracking for new packages, but syncing group membership is automatic.
Fabio
Dne 25. 08. 22 v 16:38 Fabio Valentini napsal(a):
On Thu, Aug 25, 2022 at 4:34 PM Ankur Sinha sanjay.ankur@gmail.com wrote:
On Thu, Aug 25, 2022 15:58:12 +0200, Fabio Valentini wrote:
Yeah that should be pretty simple. Get list of source packages in $release, paste it into the "Add packages" page, set release combobox to $release, submit. I do that regularly for all Rust packages.
The rust-sig packages are also set to automatically be tracked by Koschei from the looks of it and I expect this can then be done for all packages instead of for individual SIGs:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/kosc...
As far as I understand, this only automatically adds packages that are associated with the "@rust-sig" group on dist-git to the "rust-sig" group on koschei. We still need to manually enable tracking for new packages, but syncing group membership is automatic.
https://pagure.io/fedora-infra/ansible/blob/c2c063d22fe9a36b167fd27496aa1bd8...
https://github.com/fedora-infra/koschei/blob/master/bin/koschei-track-group
Checking these two ^^ I'd say that the whole group is automatically tracked.
Vít
Fabio _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, Aug 26, 2022 at 10:09 AM Vít Ondruch vondruch@redhat.com wrote:
Dne 25. 08. 22 v 16:38 Fabio Valentini napsal(a):
On Thu, Aug 25, 2022 at 4:34 PM Ankur Sinha sanjay.ankur@gmail.com wrote:
On Thu, Aug 25, 2022 15:58:12 +0200, Fabio Valentini wrote:
Yeah that should be pretty simple. Get list of source packages in $release, paste it into the "Add packages" page, set release combobox to $release, submit. I do that regularly for all Rust packages.
The rust-sig packages are also set to automatically be tracked by Koschei from the looks of it and I expect this can then be done for all packages instead of for individual SIGs:
https://pagure.io/fedora-infra/ansible/blob/main/f/roles/openshift-apps/kosc...
As far as I understand, this only automatically adds packages that are associated with the "@rust-sig" group on dist-git to the "rust-sig" group on koschei. We still need to manually enable tracking for new packages, but syncing group membership is automatic.
https://pagure.io/fedora-infra/ansible/blob/c2c063d22fe9a36b167fd27496aa1bd8...
https://github.com/fedora-infra/koschei/blob/master/bin/koschei-track-group
Checking these two ^^ I'd say that the whole group is automatically tracked.
That is correct. All packages for which rust-sig dist-git group is a watcher (configurable in dist-git) are added to rust-sig group in Koschei. Then all packages that belong to the rust-sig group in Koschei are set as tracked. This is done by cron jobs scheduled every 3 hours.
Anyone can request packages meeting certain criteria (eg. name matching some pattern, or owned/watched by a particular person/group) to be auto-tracked by Koschei or added to a particular group of packages.
-- Mikolaj Izdebski
Vít
Fabio _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Thu, Aug 25 2022 at 09:53:49 AM +0200, Miro Hrončok mhroncok@redhat.com wrote:
during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically.
Yay, this should have been done long ago. ;)
On Thu, 2022-08-25 at 09:53 +0200, Miro Hrončok wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
- Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
- One-time enablement of all existing packages
That should be doable. Right?
- Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
Can we do this? How can I help?
Big +1. Even for 1), if it turns out things burn, we now have a great case to ask for more resources - we're not just theoretically trying to bump things up against possible capacity requirements, we need to fix something that's a problem right now! ;)
On Thu, Aug 25, 2022 at 7:57 AM Miro Hrončok mhroncok@redhat.com wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically.
I am not anywhere near the deciders group, but this (enabling Koschei by default) just makes too much sense not to do for the vast majority of cases (I have no doubt there are edge cases (there are always edge cases) where disabling Koschei is the right choice, too).
And while I am a small time packager, I enable Koschei for all my packages (CI is not a panacea, of course, no matter what some advocates might try to assert).
On Thu, 25 Aug 2022 at 03:58, Miro Hrončok mhroncok@redhat.com wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
- Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
It will, but we will deal with it. Koschei builds have had to be throttled down multiple times in the past due to the fact that all those builds working or failed take up some disk space on builders for a bit. Get a lot of churn and koschei builds and you end up with builders without disk space and various developers who are saying yes! in the past yelling about not being able to do their real work :). We will just have to figure out how to throttle things down safely to make it work. The current koji+all-other-stuff is a batch operating system (line up the tasks and do them until they are done) which we treat like a time-share which only holds when things are 'low' usage.
Long term I think we are going to need to figure out ways to get the fedora buildsystem safely and securely into various clouds to allow for many more builders and resources. [This isn't just koji, but the assumptions that all the other tools in the fedora system from the 'dashboard' to the koji+bodhi+osbs+odcs+pungi+etc+etc build system that all the resources are close and local in a 'sealed' firewalled off space. ]
Dne 26. 08. 22 v 15:22 Stephen Smoogen napsal(a):
On Thu, 25 Aug 2022 at 03:58, Miro Hrončok mhroncok@redhat.com wrote:
Hello folks, during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically. There seem to be a consensus by FESCo members, that it would be a good thing. What would it take? 1) Koji resources I think we can try to enable this and see if it burns. I think ti won't.
It will, but we will deal with it. Koschei builds have had to be throttled down multiple times in the past due to the fact that all those builds working or failed take up some disk space on builders for a bit.
Unfortunately this took toll on Koschei usability, because the logs are not preserved :/ I wish the logs were extracted from Koji and stored somewhere where they can be kept for more then two weeks.
Just yesterday, I was looking into rubygem-rubyzip failures [1], which happened every ~8th build. Without logs, Koschei was not really helpful.
Vít
P.S. for the curious, while the package is noarch, the build failed when executed on s390x. IOW LE / BE issue.
[1] https://koschei.fedoraproject.org/package/rubygem-rubyzip
On 8/26/22 09:22, Stephen Smoogen wrote:
On Thu, 25 Aug 2022 at 03:58, Miro Hrončok mhroncok@redhat.com wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
- Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
It will, but we will deal with it. Koschei builds have had to be throttled down multiple times in the past due to the fact that all those builds working or failed take up some disk space on builders for a bit. Get a lot of churn and koschei builds and you end up with builders without disk space and various developers who are saying yes! in the past yelling about not being able to do their real work :). We will just have to figure out how to throttle things down safely to make it work. The current koji+all-other-stuff is a batch operating system (line up the tasks and do them until they are done) which we treat like a time-share which only holds when things are 'low' usage.
Long term I think we are going to need to figure out ways to get the fedora buildsystem safely and securely into various clouds to allow for many more builders and resources. [This isn't just koji, but the assumptions that all the other tools in the fedora system from the 'dashboard' to the koji+bodhi+osbs+odcs+pungi+etc+etc build system that all the resources are close and local in a 'sealed' firewalled off space. ]
How bursty is the builder workload? Also, if you go this route, I strongly recommend enforcing CA pinning (not public key pinning!) to the appropriate Google/Amazon roots.
On Thu, Aug 25, 2022 at 9:53 AM Miro Hrončok mhroncok@redhat.com wrote:
Hello folks,
during our Nest FESCo session, we've talked about enabling Koschei [1] for all packages automatically.
There seem to be a consensus by FESCo members, that it would be a good thing.
What would it take?
- Koji resources
I think we can try to enable this and see if it burns. I think ti won't.
I don't expect Koschei would use much more Koji resources than it already does.
Koschei maintains a priority queue of package rebuilds and submits scratch builds as available Koji resources allow. The queue is almost never empty - there are almost always some packages to rebuild. An increased number of tracked packages would simply grow the queue, but would not lead to submitting much more scratch builds. IOW, packages would simply be rebuilt less often.
To submit more scratch builds we would need larger builder capacity. This doesn't necessarily mean more or better hardware. Better Koji configuration would help a lot. We have some very powerful builders with up to 224 processors, but their capacity is set to 2. This means that the builder stops accepting new tasks once load gets to 2, which is less than 1 %.
Example buildhw-a64-20.iad2.fedoraproject.org Capacity is 2, check with: koji hostinfo buildhw-a64-20.iad2.fedoraproject.org 224 CPUs, load average: 2.04, 2.07, 2.05 memory: 251Gi total, 7.3Gi used, 242Gi available Yet, at the time of writing the builder is marked as not ready (!!) for taking more builds due to exceeded capacity.
This is not an individual case, we have many builders like that.
Making all packages as tracked without increasing Koji builder capacity will limit Koschei usefulness for packagers that care about Koschei enough to manually opt-in.
- One-time enablement of all existing packages
That should be doable. Right?
- Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
I can implement points 2 and 3 easily, as long as there is consensus to do so.
-- Mikolaj Izdebski
Can we do this? How can I help?
[1] https://fedoraproject.org/wiki/Koschei
Miro Hrončok
Phone: +420777974800 IRC: mhroncok
On Tue, 30 Aug 2022 at 14:16, Mikolaj Izdebski mizdebsk@redhat.com wrote:
On Thu, Aug 25, 2022 at 9:53 AM Miro Hrončok mhroncok@redhat.com wrote:
To submit more scratch builds we would need larger builder capacity. This doesn't necessarily mean more or better hardware. Better Koji configuration would help a lot. We have some very powerful builders with up to 224 processors, but their capacity is set to 2. This means that the builder stops accepting new tasks once load gets to 2, which is less than 1 %.
Example buildhw-a64-20.iad2.fedoraproject.org Capacity is 2, check with: koji hostinfo buildhw-a64-20.iad2.fedoraproject.org 224 CPUs, load average: 2.04, 2.07, 2.05 memory: 251Gi total, 7.3Gi used, 242Gi available Yet, at the time of writing the builder is marked as not ready (!!) for taking more builds due to exceeded capacity.
This is not an individual case, we have many builders like that.
I believe there are 2 builders with 224 CPU's. They are both 'prototype' aarch64 systems we got on loan from a vendor and are tempermental. They do not have good disk IO and do not have the capability of increasing the disk IO. This means that you have a very very fast cpu and slow as treacle disk io. The network on them has also been 'fun' where if the net access got too high, it would require a complete power cycle with the plugs pulled out because the BMC is not licensed.
Most of the 'big' builders I remember are similar in hardware.. They look great on paper, but if you try to give them a large capacity you end up with even slower builds because they are loaners or spares which are good for specific workloads and not general usage.
On Wed, Aug 31, 2022 at 3:57 PM Stephen Smoogen ssmoogen@redhat.com wrote:
On Tue, 30 Aug 2022 at 14:16, Mikolaj Izdebski mizdebsk@redhat.com wrote:
On Thu, Aug 25, 2022 at 9:53 AM Miro Hrončok mhroncok@redhat.com wrote:
To submit more scratch builds we would need larger builder capacity. This doesn't necessarily mean more or better hardware. Better Koji configuration would help a lot. We have some very powerful builders with up to 224 processors, but their capacity is set to 2. This means that the builder stops accepting new tasks once load gets to 2, which is less than 1 %.
Example buildhw-a64-20.iad2.fedoraproject.org Capacity is 2, check with: koji hostinfo buildhw-a64-20.iad2.fedoraproject.org 224 CPUs, load average: 2.04, 2.07, 2.05 memory: 251Gi total, 7.3Gi used, 242Gi available Yet, at the time of writing the builder is marked as not ready (!!) for taking more builds due to exceeded capacity.
This is not an individual case, we have many builders like that.
I believe there are 2 builders with 224 CPU's. They are both 'prototype' aarch64 systems we got on loan from a vendor and are tempermental. They do not have good disk IO and do not have the capability of increasing the disk IO. This means that you have a very very fast cpu and slow as treacle disk io. The network on them has also been 'fun' where if the net access got too high, it would require a complete power cycle with the plugs pulled out because the BMC is not licensed.
Slow disk could be worked around by using tmpfs for mock chroots - there is lots of free memory. I was not aware of the networking issues.
-- Mikolaj Izdebski
Most of the 'big' builders I remember are similar in hardware.. They look great on paper, but if you try to give them a large capacity you end up with even slower builds because they are loaners or spares which are good for specific workloads and not general usage.
-- Stephen Smoogen, Red Hat Automotive Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On 30. 08. 22 20:15, Mikolaj Izdebski wrote:
- One-time enablement of all existing packages
That should be doable. Right?
- Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
I can implement points 2 and 3 easily, as long as there is consensus to do so.
I plan to propose a Fedora Linux 38 change proposal for this to get consensus. Should I list you as a change owner?
On Wed, Aug 31, 2022 at 4:03 PM Miro Hrončok mhroncok@redhat.com wrote:
On 30. 08. 22 20:15, Mikolaj Izdebski wrote:
- One-time enablement of all existing packages
That should be doable. Right?
- Automatic enablement of all new packages
That should be just a matter of changing the defaults. Correct?
I can implement points 2 and 3 easily, as long as there is consensus to do so.
I plan to propose a Fedora Linux 38 change proposal for this to get consensus. Should I list you as a change owner?
I don't think the Change process is needed for this infrastructure change. But if you like to, feel free to. I can be a co-owner if you like.
My plan was to bring this topic to the infrastructure list and gather more feedback there. Unless there were any objections I would proceed to implement the proposal after Fedora 37 beta freeze - end of September. I would rather not implement this during infra freeze as this change is likely to put more load on services Koschei depends on.
-- Mikolaj Izdebski
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok