On 07/22/2010 05:44 AM, Dan HorĂ¡k wrote:
Hi all,
there are some changes that needs to be done on the Fedora/s390x infrastructure for various reasons and we should prepare a plan how and when to do them.
Tasks
- upgrade builders to something with kernel 2.6.32+ like EL-6 beta
Yesterday a new glibc was built in primary Fedora, it drops some compatibility stuff for old kernels and now requires kernel 2.6.32+ on the builders.
- redesign resource allocation for the builders
The builders (and the squid cache) don't behave very well when under full load. Timeouts when downloading the buildroots thru the cache are quite rare (but they happen every day). Other observed behaviour looks like a completely swapped out guest, it doesn't respond to ping, koji doesn't update its status on the hub and it can take minutes, maybe even tens of minutes, before the machine starts responding again. Unfortunately often this behaviour means stuck builds and manually restarting the builder daemon. I think part of this issue could be solved on the z/VM side (size of RAM, number of CPUs per builder), part could be tuning the koji configuration (max jobs per builder, max load, parallel make, ...). Also interesting would be to see performance/resource usage statistics from z/VM.
I'm open to suggestion I believe everyone has the specs on the build lpar. If by some chance we need more memory/storage, we will need to plead our case with Arlinton Bourne. If we are in need of disks for VM paging volumes (I just added two more not long ago), I only have a couple 3390's left available on that lpar.
- rebuild the storage on the hub
The sub-optimal storage configuration is known for some time, but there is still room on the disks for few months of work (my guess).
During my backups of the hub, I have been contacted. I will forward the message to relevant parties off-list.
- upgrade Koji to 1.4
Here we should be on par with the primary Fedora buildsystem.
Timing
- ASAP, let's say till end of next week, because it blocks building
packages that were built after July 21 in primary koji 2. together with 1. 3. till end of August, earlier date means fewer data to sync 4. together with 3
Outages
- none, can be done one builder at a time
- yes
- yes
- yes
Comments, ideas, corrections, etc. are welcome
Dan
s390x mailing list s390x@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/s390x