Hey all,
Lately, I've been using the COPR ARM builders a fair bit: I've been working on Fedora Asahi stuff with Davide Cavalca for the past several months, and I've noticed that builds in COPR are easily at least 2x slower than their equivalent builds in Koji, most of the time considerably worse.
For a concrete example: I've been building kernel-asahi in ngompa/fedora-asahi and ngompa/fedora-asahi-dev COPRs for some time now, and each kernel build is basically 3 hours. Most recently, I built a kernel based on 6.1-rc5 (https://copr.fedorainfracloud.org/coprs/ngompa/fedora-asahi-dev/build/504189...) that took 3 hours. The equivalent build in Koji (https://koji.fedoraproject.org/koji/taskinfo?taskID=94172035) took less than 1.25 hours.
I tried earlier today to build QtWebEngine in COPR for testing a fix, and it couldn't complete in the 5 hour time limit it has by default. I eventually did a build locally (which took about an hour to build) and validated it before pushing it to Koji (https://koji.fedoraproject.org/koji/taskinfo?taskID=94372924), which took a little under 3.15 hours.
I thought we had access to super-fast Graviton-based builders for COPR. Our builds should be *flying* compared to Koji or my local resources. So what's going on?
-- 真実はいつも一つ!/ Always, there's only one truth!
On pondělí 21. listopadu 2022 3:16:39 CET Neal Gompa wrote:
Hey all,
Lately, I've been using the COPR ARM builders a fair bit: I've been working on Fedora Asahi stuff with Davide Cavalca for the past several months, and I've noticed that builds in COPR are easily at least 2x slower than their equivalent builds in Koji, most of the time considerably worse.
For a concrete example: I've been building kernel-asahi in ngompa/fedora-asahi and ngompa/fedora-asahi-dev COPRs for some time now, and each kernel build is basically 3 hours. Most recently, I built a kernel based on 6.1-rc5 (https://copr.fedorainfracloud.org/coprs/ngompa/fedora-asahi-dev/build/504189...) that took 3 hours. The equivalent build in Koji (https://koji.fedoraproject.org/koji/taskinfo?taskID=94172035) took less than 1.25 hours.
I tried earlier today to build QtWebEngine in COPR for testing a fix, and it couldn't complete in the 5 hour time limit it has by default. I eventually did a build locally (which took about an hour to build) and validated it before pushing it to Koji (https://koji.fedoraproject.org/koji/taskinfo?taskID=94372924), which took a little under 3.15 hours.
I thought we had access to super-fast Graviton-based builders for COPR. Our builds should be *flying* compared to Koji or my local resources. So what's going on?
Not sure what is in Koji, so just some facts about the Copr arm builders...
Copr uses 'a1.2xlarge' machines: https://aws.amazon.com/ec2/instance-types/a1/
Copr uses mock's tmpfs feature, with cca 140GB SWAP volume on a gp2 disk: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-purpose.html
Pavel
-- 真実はいつも一つ!/ Always, there's only one truth!
On pondělí 21. listopadu 2022 3:16:39 CET Neal Gompa wrote:
Not sure what is in Koji, so just some facts about the Copr arm builders...
Copr uses 'a1.2xlarge' machines: https://aws.amazon.com/ec2/instance-types/a1/
We do have access to the later generations: The [CMR]6g and then the latest Graviton 3 instances in the C7g family.
Our (AWS’s) engineering team highly discourages the use of the a1 instance types at this point. Can we move that to standard T4g.2xl or better yet update that to a C6g.2xl?
It sounds like we need to update the infrastructure definitions to include the later versions. These next-generation instance types will definitely increase the efficiency of the workload.
Copr uses mock's tmpfs feature, with cca 140GB SWAP volume on a gp2 disk: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-purpose.html
Some of the justification should come directly from the ec2 optimizer in the account. We should probably take a look at the profile information there to determine what will be the best fit cost/performance wise.
On úterý 22. listopadu 2022 12:23:54 CET David Duncan wrote:
On pondělí 21. listopadu 2022 3:16:39 CET Neal Gompa wrote:
Not sure what is in Koji, so just some facts about the Copr arm builders...
Copr uses 'a1.2xlarge' machines: https://aws.amazon.com/ec2/instance-types/a1/
We do have access to the later generations: The [CMR]6g and then the latest Graviton 3 instances in the C7g family.
Our (AWS’s) engineering team highly discourages the use of the a1 instance types at this point. Can we move that to standard T4g.2xl or better yet update that to a C6g.2xl?
I tested c6g and c7g today.
They seem to work, but subscription-manager seems to fail (some entitlement issue). When we sort this out, it would be an easy switch, thank you for the tip!
It sounds like we need to update the infrastructure definitions to include the later versions. These next-generation instance types will definitely increase the efficiency of the workload.
They indeed are much faster.
Copr uses mock's tmpfs feature, with cca 140GB SWAP volume on a gp2 disk: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-purpose.html
Some of the justification should come directly from the ec2 optimizer in the account. We should probably take a look at the profile information there to determine what will be the best fit cost/performance wise.
Never used ec2 optimizer. Another very good tip, thanks.
Pavel
On úterý 22. listopadu 2022 19:21:01 CET Pavel Raiskup wrote:
On úterý 22. listopadu 2022 12:23:54 CET David Duncan wrote:
On pondělí 21. listopadu 2022 3:16:39 CET Neal Gompa wrote:
Not sure what is in Koji, so just some facts about the Copr arm builders...
Copr uses 'a1.2xlarge' machines: https://aws.amazon.com/ec2/instance-types/a1/
We do have access to the later generations: The [CMR]6g and then the latest Graviton 3 instances in the C7g family.
Our (AWS’s) engineering team highly discourages the use of the a1 instance types at this point. Can we move that to standard T4g.2xl or better yet update that to a C6g.2xl?
I tested c6g and c7g today.
Note this is being tested on staging instance, with c7g.xlarge: https://copr.stg.fedoraproject.org/coprs/praiskup/ping/build/2910412/ So the CPU count and RAM size stays:
https://pagure.io/fedora-infra/ansible/c/5023fd514222eb77e6b1fc0c3494aa5a8e7...
If everything goes OK, we could have it in production next week.
Pavel
They seem to work, but subscription-manager seems to fail (some entitlement issue). When we sort this out, it would be an easy switch, thank you for the tip!
It sounds like we need to update the infrastructure definitions to include the later versions. These next-generation instance types will definitely increase the efficiency of the workload.
They indeed are much faster.
Copr uses mock's tmpfs feature, with cca 140GB SWAP volume on a gp2 disk: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/general-purpose.html
Some of the justification should come directly from the ec2 optimizer in the account. We should probably take a look at the profile information there to determine what will be the best fit cost/performance wise.
Never used ec2 optimizer. Another very good tip, thanks.
Pavel
On Sun, 20 Nov 2022 at 21:18, Neal Gompa ngompa13@gmail.com wrote:
Hey all,
Lately, I've been using the COPR ARM builders a fair bit: I've been working on Fedora Asahi stuff with Davide Cavalca for the past several months, and I've noticed that builds in COPR are easily at least 2x slower than their equivalent builds in Koji, most of the time considerably worse.
For a concrete example: I've been building kernel-asahi in ngompa/fedora-asahi and ngompa/fedora-asahi-dev COPRs for some time now, and each kernel build is basically 3 hours. Most recently, I built a kernel based on 6.1-rc5 ( https://copr.fedorainfracloud.org/coprs/ngompa/fedora-asahi-dev/build/504189... ) that took 3 hours. The equivalent build in Koji (https://koji.fedoraproject.org/koji/taskinfo?taskID=94172035) took less than 1.25 hours.
I tried earlier today to build QtWebEngine in COPR for testing a fix, and it couldn't complete in the 5 hour time limit it has by default. I eventually did a build locally (which took about an hour to build) and validated it before pushing it to Koji (https://koji.fedoraproject.org/koji/taskinfo?taskID=94372924), which took a little under 3.15 hours.
I thought we had access to super-fast Graviton-based builders for COPR. Our builds should be *flying* compared to Koji or my local resources. So what's going on?
I have been noticing a sizeable slowdown in builds on Graviton in other places unless the hardware is a dedicated hardware instance. My guess is that ARM is no longer niche and a lot of other teams are using them which means we are getting 'swapped in and out' in the cloud with everyone else.
buildsys@lists.fedoraproject.org