I built the latest ceph-14 (14.2.2) on rawhide successfully two days ago.
Two different builds on f30 built or are building fine on x86_64, i686, and aarch64, but failed with different errors on ppc64le at different places in the build. One looks like it ran out of space in the file system. The other may have been OOM killed (?).
https://kojipkgs.fedoraproject.org//work/tasks/2448/36422448/build.log
https://kojipkgs.fedoraproject.org//work/tasks/4819/36444819/build.log
Thanks,
--
Kaleb
"KK" == Kaleb Keithley kkeithle@redhat.com writes:
KK> I built the latest ceph-14 (14.2.2) on rawhide successfully two days KK> ago. Two different builds on f30 built or are building fine on KK> x86_64, i686, and aarch64, but failed with different errors on KK> ppc64le at different places in the build. One looks like it ran out KK> of space in the file system. The other may have been OOM killed (?).
There was just a bit of talk about this in IRC. The issue seems to be that the CPU count of the PPC64le builders was bumped from 4 to 12, but the amount of memory was unchanged at 10GB RAM/2GB swap. This could potentially cause resource exhaustion.
Seems they've now been bumped to 22GB of RAM, which should help with OOM issues but probably not with disk space issues.
- J<
On 7/23/19 11:36 AM, Jason L Tibbitts III wrote:
"KK" == Kaleb Keithley kkeithle@redhat.com writes:
KK> I built the latest ceph-14 (14.2.2) on rawhide successfully two days KK> ago. Two different builds on f30 built or are building fine on KK> x86_64, i686, and aarch64, but failed with different errors on KK> ppc64le at different places in the build. One looks like it ran out KK> of space in the file system. The other may have been OOM killed (?).
There was just a bit of talk about this in IRC. The issue seems to be that the CPU count of the PPC64le builders was bumped from 4 to 12, but the amount of memory was unchanged at 10GB RAM/2GB swap. This could potentially cause resource exhaustion.
Seems they've now been bumped to 22GB of RAM, which should help with OOM issues but probably not with disk space issues.
Right. Please file a ticket or let us know if you hit the disk issue again.
kevin
On 07/23/2019 11:36 AM, Jason L Tibbitts III wrote:
"KK" == Kaleb Keithley kkeithle@redhat.com writes:
KK> I built the latest ceph-14 (14.2.2) on rawhide successfully two days KK> ago. Two different builds on f30 built or are building fine on KK> x86_64, i686, and aarch64, but failed with different errors on KK> ppc64le at different places in the build. One looks like it ran out KK> of space in the file system. The other may have been OOM killed (?).
There was just a bit of talk about this in IRC. The issue seems to be that the CPU count of the PPC64le builders was bumped from 4 to 12, but the amount of memory was unchanged at 10GB RAM/2GB swap. This could potentially cause resource exhaustion.
Seems they've now been bumped to 22GB of RAM, which should help with OOM issues but probably not with disk space issues.
Are these updated builders only used for f30? Because I'm still getting builders with 4 CPU/ 10GB RAM/2GB swap on rawhide. For example: https://koji.fedoraproject.org/koji/taskinfo?taskID=36476090
-Tom
- J<
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 7/24/19 3:53 PM, Tom Stellard wrote:
On 07/23/2019 11:36 AM, Jason L Tibbitts III wrote:
> "KK" == Kaleb Keithley kkeithle@redhat.com writes:
KK> I built the latest ceph-14 (14.2.2) on rawhide successfully two days KK> ago. Two different builds on f30 built or are building fine on KK> x86_64, i686, and aarch64, but failed with different errors on KK> ppc64le at different places in the build. One looks like it ran out KK> of space in the file system. The other may have been OOM killed (?).
There was just a bit of talk about this in IRC. The issue seems to be that the CPU count of the PPC64le builders was bumped from 4 to 12, but the amount of memory was unchanged at 10GB RAM/2GB swap. This could potentially cause resource exhaustion.
Seems they've now been bumped to 22GB of RAM, which should help with OOM issues but probably not with disk space issues.
Are these updated builders only used for f30? Because I'm still getting builders with 4 CPU/ 10GB RAM/2GB swap on rawhide. For example: https://koji.fedoraproject.org/koji/taskinfo?taskID=36476090
It's not all ppc64le builders. Only the ones on the power9 virthosts for now (01-19). I'm planning on redoing the rest (20-29) (which are on power8 vhosts), but I ran out of time before the mass rebuild. I'll do them as soon as it's over, likely next week.
kevin
"TS" == Tom Stellard tstellar@redhat.com writes:
TS> Are these updated builders only used for f30?
It appears that there are 29 PPC64le builders configured currently: https://koji.fedoraproject.org/koji/hosts?start=80&state=enabled&ord...
They don't all have the same "capacity" rating.
TS> Because I'm still getting builders with 4 CPU/ 10GB RAM/2GB swap on TS> rawhide.
I imagine that there is some randomness in play. The build you list ran on buildvm-ppc64le-21.ppc.fedoraproject.org which has a capacity rating of 2.0. Some of the builders have a rating of 4.0. (Which I guess doesn't correspond to the increase in core count, but I don't know how it's calculated.)
- J<
Jason L Tibbitts III tibbs@math.uh.edu writes:
"KK" == Kaleb Keithley kkeithle@redhat.com writes:
KK> I built the latest ceph-14 (14.2.2) on rawhide successfully two days KK> ago. Two different builds on f30 built or are building fine on KK> x86_64, i686, and aarch64, but failed with different errors on KK> ppc64le at different places in the build. One looks like it ran out KK> of space in the file system. The other may have been OOM killed (?).
There was just a bit of talk about this in IRC. The issue seems to be that the CPU count of the PPC64le builders was bumped from 4 to 12, but the amount of memory was unchanged at 10GB RAM/2GB swap. This could potentially cause resource exhaustion.
Seems they've now been bumped to 22GB of RAM, which should help with OOM issues but probably not with disk space issues.
Although not directly related, but I thought I'll throw this in: openSUSE has a handy package called memory-constraints which features a single macro: %limit_build.
You give this macro the amount of RAM you expect make/ninja/$my_build_cmd to consume at peak per thread and it sets the number of jobs via overriding _smp_mflags so that you won't OOM.
Source: https://build.opensuse.org/package/view_file/network:chromium/memory-constra...
That could be useful for arches which have builders with a lot of cores but not a lot of RAM or builds which consume excessive amounts.
Cheers,
Dan
On Fri, 26 Jul 2019 09:02:57 +0200 Dan Čermák dan.cermak@cgc-instruments.com wrote:
Jason L Tibbitts III tibbs@math.uh.edu writes:
> "KK" == Kaleb Keithley kkeithle@redhat.com writes:
KK> I built the latest ceph-14 (14.2.2) on rawhide successfully two KK> days ago. Two different builds on f30 built or are building KK> fine on x86_64, i686, and aarch64, but failed with different KK> errors on ppc64le at different places in the build. One looks KK> like it ran out of space in the file system. The other may have KK> been OOM killed (?).
There was just a bit of talk about this in IRC. The issue seems to be that the CPU count of the PPC64le builders was bumped from 4 to 12, but the amount of memory was unchanged at 10GB RAM/2GB swap. This could potentially cause resource exhaustion.
Seems they've now been bumped to 22GB of RAM, which should help with OOM issues but probably not with disk space issues.
Although not directly related, but I thought I'll throw this in: openSUSE has a handy package called memory-constraints which features a single macro: %limit_build.
You give this macro the amount of RAM you expect make/ninja/$my_build_cmd to consume at peak per thread and it sets the number of jobs via overriding _smp_mflags so that you won't OOM.
Source: https://build.opensuse.org/package/view_file/network:chromium/memory-constra...
That could be useful for arches which have builders with a lot of cores but not a lot of RAM or builds which consume excessive amounts.
long time ago I filed a bug/RFE to compute the "-j" value from the number of cpus and available memory, still waiting on me to actually implement it ;-)
Dan