On Sun, 20 Nov 2022 at 21:18, Neal Gompa ngompa13@gmail.com wrote:
Hey all,
Lately, I've been using the COPR ARM builders a fair bit: I've been working on Fedora Asahi stuff with Davide Cavalca for the past several months, and I've noticed that builds in COPR are easily at least 2x slower than their equivalent builds in Koji, most of the time considerably worse.
For a concrete example: I've been building kernel-asahi in ngompa/fedora-asahi and ngompa/fedora-asahi-dev COPRs for some time now, and each kernel build is basically 3 hours. Most recently, I built a kernel based on 6.1-rc5 ( https://copr.fedorainfracloud.org/coprs/ngompa/fedora-asahi-dev/build/504189... ) that took 3 hours. The equivalent build in Koji (https://koji.fedoraproject.org/koji/taskinfo?taskID=94172035) took less than 1.25 hours.
I tried earlier today to build QtWebEngine in COPR for testing a fix, and it couldn't complete in the 5 hour time limit it has by default. I eventually did a build locally (which took about an hour to build) and validated it before pushing it to Koji (https://koji.fedoraproject.org/koji/taskinfo?taskID=94372924), which took a little under 3.15 hours.
I thought we had access to super-fast Graviton-based builders for COPR. Our builds should be *flying* compared to Koji or my local resources. So what's going on?
I have been noticing a sizeable slowdown in builds on Graviton in other places unless the hardware is a dedicated hardware instance. My guess is that ARM is no longer niche and a lot of other teams are using them which means we are getting 'swapped in and out' in the cloud with everyone else.