On Sun, 20 Nov 2022 at 21:18, Neal Gompa <ngompa13@gmail.com> wrote:
Hey all,

Lately, I've been using the COPR ARM builders a fair bit: I've been
working on Fedora Asahi stuff with Davide Cavalca for the past several
months, and I've noticed that builds in COPR are easily at least 2x
slower than their equivalent builds in Koji, most of the time
considerably worse.

For a concrete example: I've been building kernel-asahi in
ngompa/fedora-asahi and ngompa/fedora-asahi-dev COPRs for some time
now, and each kernel build is basically 3 hours. Most recently, I
built a kernel based on 6.1-rc5
(https://copr.fedorainfracloud.org/coprs/ngompa/fedora-asahi-dev/build/5041891/)
that took 3 hours. The equivalent build in Koji
(https://koji.fedoraproject.org/koji/taskinfo?taskID=94172035) took
less than 1.25 hours.

I tried earlier today to build QtWebEngine in COPR for testing a fix,
and it couldn't complete in the 5 hour time limit it has by default. I
eventually did a build locally (which took about an hour to build) and
validated it before pushing it to Koji
(https://koji.fedoraproject.org/koji/taskinfo?taskID=94372924), which
took a little under 3.15 hours.

I thought we had access to super-fast Graviton-based builders for
COPR. Our builds should be *flying* compared to Koji or my local
resources. So what's going on?


I have been noticing a sizeable slowdown in builds on Graviton in other places unless the hardware is a dedicated hardware instance. My guess is that ARM is no longer niche and a lot of other teams are using them which means we are getting 'swapped in and out' in the cloud with everyone else.
 
--
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren