On Mon, Jan 2, 2012 at 6:55 PM, Dennis Gilmore <dennis(a)ausil.us> wrote:
El Sat, 24 Dec 2011 23:25:56 -0600
Dennis Gilmore <dennis(a)ausil.us> escribió:
> When we were testing build were happening really fast, once we loaded
> up the build jobs things have become really slow
> at 1:19 utc and at 5:18 utc four hours later the buildrequires are
> still being installed. australia is completely io bound. i think we
> need to see how we can spread the io load. ~30 hosts reading and
> writing to the 4 spindles just saturates the disk io. i guess options
> are find a way to add more spindles. move /var/lib/mock to sdcard, see
> if we can get some kind of san that has alot of smaller fast disks.
> get some 2.5" usb drives one for each builder. some other idea? is
> there anyway we could add 4-8 more disks to australia. the size of the
> matters little. gaining more iops by adding more spindles would help.
> Just throwing out what im seeing. lets see what ideas we can come up
> with. performance is better than before. and seneca has done a great
> job and put a lot of hard work into the reorg and im thankful for
> that. We just have another bottleneck to address.
> arm mailing list
running 10 hours now. its inited the chroot and installing the srpm. it
is a big srpm but still. performance on the hfp builders while not
great is better. when looking at a sfp builder the mock process is
using 100% cpu and in a D state. there is one of two issues here. the
hfp builders are running a slightly older kernel. so it could be a
kernel ext3 nfs or loop device issue, upgrading a hfp builder for
testing would confirm it. or its some issue with decompresion of the
rpms. I do think that disk io on the nfs server is partly to blame. the
box sits at a load of 9 all the time with 30% or so io wait. Id like to
take a layer or 2 of the complexity out. switching over to nfsv4 we
should be able to run mock on it. im going to do some compression tests
with the latest 220.127.116.11 kernel.
Nearly two days after I submitted this build its still not actually to
the point of building.
This is a dire situation and doesn't give me much hope of being able
to follow closely behind mainline at this rate.