Greetings.
Since I am out on PTO all next week, I thought I would share a status of where we are with koji so others could continue the work in my absense.
We have a number of things that need updates on koji builders.
* mock 1.4.15+ is needed for dynamic buildrequires * createrepo_c 0.14.1-2+ is needed for epel8/fedora non modular builds against modules * koji build with backported patches for: ** seperate srpm repos with buildroot source repos ** bare mode repo merging needed for epel8/fedora buildroot ** dynamic buildrequires (along with newer mock) * oz and imagefactory have gone python3 in f31/rawhide, and oz isn't installable in f30 anymore.
So, I have built a koji in rawhide with all the patches we need.
I have then built in f29/f30-infra tags all the stuff we need: koji with patches createrepo_c oz imagefactory
A sticking point for updating has been that our armv7 builders are stuck on f27 due to a kernel (or perhaps userspace?) bug that causes then to pause when doing kernel/glibc/other builds. pbrobinson and pwhalen are trying to isolate this and fix it, but until then it looks like we can move them to fedora 29 + a specific older kernel. I have setup a kickstart that installs this kernel and set ansible to disallow dnf-automatic updating the kernel on them.
I've been testing with buildvm-armv7-22 in prod, and buildvm-armv7-01.stg in staging, and so far they have survived several kernel builds.
I have updated all of staging with the above packages.
What still needs doing next week:
* reinstall all the prod armv7 builders with the f29 kickstart and confirm they don't blow up. * promote createrepo_c, koji, oz, imagefactory builds from f29-infra-stg to f29-infra, and epel7-infra-stg to epel7-infra * update koji hubs and all builders to above packages. * watch for problems with oz/imagefactory/createrepo_c/koji and fix or revert. * profit
After I get back we can look at moving to f30, but for now f29 should work. Hopefully by then we will have figured out the armv7 issue and be able to move them to the latest kernel.
I might have time later tonight/tomorrow to work further on this, but not sure I will. If so, I'll update this thread. If not, it will have to wait until I get back from vacation, or someone else will need to move it forward next week.
Thanks,
kevin
infrastructure@lists.fedoraproject.org