Disk Space Issues - cvs-int/build hosts - (includes general note on Nagios Disk notifications)
Hey guys,
I seem to have deleted the nagios notification that I want to mention in particular, but as I noted in SOP/Nagios the %ages noted in the e-mails are what is left, so inode=99% doesn't mean that 99% of the inodes are used, it means 99% are still free.
Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge.
Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.
Lets take banshee for example, a package which I adopted....
$ ls -l /repo/pkgs/banshee/ total 88 drwxrwsr-x 3 apache repoextras 4096 May 3 2006 banshee-0.10.10.tar.gz drwxrwsr-x 3 apache repoextras 4096 Aug 7 2006 banshee-0.10.11.tar.gz drwxrwsr-x 3 apache repoextras 4096 Aug 24 2006 banshee-0.10.12.tar.gz drwxrwsr-x 3 apache repoextras 4096 Mar 4 2006 banshee-0.10.6.tar.gz drwxrwsr-x 3 apache repoextras 4096 Mar 7 2006 banshee-0.10.7.tar.gz drwxrwsr-x 3 apache repoextras 4096 Mar 14 2006 banshee-0.10.8.tar.gz drwxrwsr-x 3 apache repoextras 4096 Apr 16 2006 banshee-0.10.9.tar.gz drwxrwsr-x 3 apache repoextras 4096 Feb 2 2007 banshee-0.11.5.tar.bz2 drwxrwsr-x 3 apache repoextras 4096 Mar 7 2007 banshee-0.12.0.tar.bz2 drwxrwsr-x 3 apache repoextras 4096 Apr 5 2007 banshee-0.12.1.tar.bz2 drwxrwsr-x 3 apache repoextras 4096 Aug 7 2007 banshee-0.13.0.tar.bz2 drwxrwsr-x 3 apache repoextras 4096 Aug 31 2007 banshee-0.13.1.tar.bz2 drwxrwsr-x 3 apache repoextras 4096 Jan 14 15:58 banshee-0.13.2.tar.bz2 drwxrwsr-x 3 apache repoextras 4096 Apr 13 01:51 banshee-1-0.98.3.tar.bz2 drwxrwsr-x 3 apache repoextras 4096 May 10 03:41 banshee-1-0.99.1.tar.bz2
$ du -sh /repo/pkgs/banshee/ 26M /repo/pkgs/banshee/ (Approximately 2M/package) At the most there should only be 4 tar balls there (R-2, R-1, R, Rawhide), R-2 only valid for one month after R has released.
Another couple of examples: $ du -sh /repo/pkgs/kernel/ 2.6G /repo/pkgs/kernel/ (900ish tarballs dating back to 2004)
$ du -sh /repo/pkgs/kdeedu 1.4G /repo/pkgs/kdeedu (48 tarballs from KDE-3.0)
With plans such and Hans' plans of creating a 500M vegastrike data SRPM and the size growth and update schedules of some packages we are going to have these nightmares more and more frequently.
Two solutions I can think of:
Create a script, go thru ALL non dead.package's grab the tarball name from sources.list and basically create a bit of a database of what we are using, scan through /repo/pkgs and either: * Move old tarballs to some archiving system (another use for archive.fp.o?) * Delete old tarballs Or throw more diskspace at cvs-int
Even if were to only remove 15% of the tarballs there (this is a very cautious estimate of the number of stale tarballs) we could potentially reach 72% diskspace available on that mount (down from 82%) - note this is very simplistic math, in essence, we could be no better off if we only removed the small stale tarballs :).
Diskspace isn't cheap, so I like delete old tarballs, I also like this option because it's not like they disappear completely, they should be in the src.rpm's already on archive.fp.o and if we accidentally delete 1 or 2 that are still needed, well grab it from src.rpm...
This leads on to my second item...
xenbuilder2 has run out of diskspace in /, it's down to 32M, thankfully koji has disabled it so it's safe for now, but wouldn't it be nice to throw say a 50GB partition dedicated solely for /var/lib/mock & /mnt/build? Yes, yes, I know money, but once again, builds are getting bigger so 'it'd be nice'.
Just thought I'd throw these two thoughts into the open, let the discussion begin!
On Tue, 2008-05-13 at 23:55 +1200, Nigel Jones wrote:
Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge.
Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.
We can't do this. We need to keep the tarballs for packages which were released (in any form, be it rawhide, an update, or the actual release -- and probably even updates-testing) for 3 years to comply with the terms of the GPL.
Not to mention that removing them would break any ability to go back and try to bisect or find the source of a problem in an automated fashion. And yes, I've actually had to go back to ancient history to do so in the past with some packages :-/
Jeremy
Jeremy Katz wrote:
On Tue, 2008-05-13 at 23:55 +1200, Nigel Jones wrote:
Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge.
Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.
We can't do this. We need to keep the tarballs for packages which were released (in any form, be it rawhide, an update, or the actual release -- and probably even updates-testing) for 3 years to comply with the terms of the GPL.
Not to mention that removing them would break any ability to go back and try to bisect or find the source of a problem in an automated fashion. And yes, I've actually had to go back to ancient history to do so in the past with some packages :-/
We were just discussing this on #fedora-admin, the other alternative would be (as I mentioned) move them to some sort of publically available archive storage setup, with the Makefile pointing towards that instead of where we normally point.
Anyway, I'm just trying to kick the ball about, I'm sure there is a miracle solution but imo at the very least it'd be somewhat nice to unladen cvs-int of some of the burden.
- Nigel
On Wed, 2008-05-14 at 00:39 +1200, Nigel Jones wrote:
Jeremy Katz wrote:
On Tue, 2008-05-13 at 23:55 +1200, Nigel Jones wrote:
Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge.
Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.
We can't do this. We need to keep the tarballs for packages which were released (in any form, be it rawhide, an update, or the actual release -- and probably even updates-testing) for 3 years to comply with the terms of the GPL.
Not to mention that removing them would break any ability to go back and try to bisect or find the source of a problem in an automated fashion. And yes, I've actually had to go back to ancient history to do so in the past with some packages :-/
We were just discussing this on #fedora-admin, the other alternative would be (as I mentioned) move them to some sort of publically available archive storage setup, with the Makefile pointing towards that instead of where we normally point.
If we're going to do that, we might as well use that for the tarball repo always and not have it special cased for old vs not. At which point, the "easy" answer is just to nfs mount that storage as /repo/pkgs :)
In fact, maybe that is the easy answer -- maybe we should just allocate some of the space off of nfs1. We're at below 200G of usage for /repo/pkgs -- if we allocated half a terabyte, it wouldn't be that big of a hit to what's available to koji and it would give us quite a bit of room to grow for /repo/pkgs.
Jeremy
Jeremy Katz wrote:
On Wed, 2008-05-14 at 00:39 +1200, Nigel Jones wrote:
Jeremy Katz wrote:
On Tue, 2008-05-13 at 23:55 +1200, Nigel Jones wrote:
Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge.
Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.
We can't do this. We need to keep the tarballs for packages which were released (in any form, be it rawhide, an update, or the actual release -- and probably even updates-testing) for 3 years to comply with the terms of the GPL.
Not to mention that removing them would break any ability to go back and try to bisect or find the source of a problem in an automated fashion. And yes, I've actually had to go back to ancient history to do so in the past with some packages :-/
We were just discussing this on #fedora-admin, the other alternative would be (as I mentioned) move them to some sort of publically available archive storage setup, with the Makefile pointing towards that instead of where we normally point.
If we're going to do that, we might as well use that for the tarball repo always and not have it special cased for old vs not. At which point, the "easy" answer is just to nfs mount that storage as /repo/pkgs :)
In fact, maybe that is the easy answer -- maybe we should just allocate some of the space off of nfs1. We're at below 200G of usage for /repo/pkgs -- if we allocated half a terabyte, it wouldn't be that big of a hit to what's available to koji and it would give us quite a bit of room to grow for /repo/pkgs.
You know, that's quite sane and I really like that idea. :)
- Nigel
Nigel Jones (dev@nigelj.com) said:
Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.
Until we get something like correspondingsource up and running, we don't have a good mechanism for weeding out only the tarballs that correspond to builds that were not/are not shipped.
Bill
On Tuesday 13 May 2008, Nigel Jones wrote:
Hey guys,
I seem to have deleted the nagios notification that I want to mention in particular, but as I noted in SOP/Nagios the %ages noted in the e-mails are what is left, so inode=99% doesn't mean that 99% of the inodes are used, it means 99% are still free.
Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge.
Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.
<snip>
We already have plans to move the lookaside cache to the netapp. at least that was the last plan i was aware of.
Diskspace isn't cheap, so I like delete old tarballs, I also like this option because it's not like they disappear completely, they should be in the src.rpm's already on archive.fp.o and if we accidentally delete 1 or 2 that are still needed, well grab it from src.rpm...
This leads on to my second item...
xenbuilder2 has run out of diskspace in /, it's down to 32M, thankfully koji has disabled it so it's safe for now, but wouldn't it be nice to throw say a 50GB partition dedicated solely for /var/lib/mock & /mnt/build? Yes, yes, I know money, but once again, builds are getting bigger so 'it'd be nice'.
xenbuilder1 and hammer2 are the oldest boxes we have in the buildsystem. closely followed by ppc1 xenbuilder2 is also quite an old box now. im cleaning up /var/lib/mock on xenbuilder2 now. failed builds dont get cleaned up automatically. we probably should reap them more often.
Dennis
On Tue, 2008-05-13 at 08:58 -0500, Dennis Gilmore wrote:
On Tuesday 13 May 2008, Nigel Jones wrote:
Hey guys,
I seem to have deleted the nagios notification that I want to mention in particular, but as I noted in SOP/Nagios the %ages noted in the e-mails are what is left, so inode=99% doesn't mean that 99% of the inodes are used, it means 99% are still free.
Anyway, what this means, is that when nagios has been complaining about cvs-int recently, in particular the fact that /git has reached WARNING. After a bit of hunting around, I found /repo/pkgs using 168GiB of the 192GiB available, which is understandable, Fedora has got huge.
Problem here, is that there are a LOT of old tarballs in that folder, which leaves me wondering if we should do a spring clean ~1 mo after release.
<snip>
We already have plans to move the lookaside cache to the netapp. at least that was the last plan i was aware of.
Diskspace isn't cheap, so I like delete old tarballs, I also like this option because it's not like they disappear completely, they should be in the src.rpm's already on archive.fp.o and if we accidentally delete 1 or 2 that are still needed, well grab it from src.rpm...
This leads on to my second item...
xenbuilder2 has run out of diskspace in /, it's down to 32M, thankfully koji has disabled it so it's safe for now, but wouldn't it be nice to throw say a 50GB partition dedicated solely for /var/lib/mock & /mnt/build? Yes, yes, I know money, but once again, builds are getting bigger so 'it'd be nice'.
xenbuilder1 and hammer2 are the oldest boxes we have in the buildsystem. closely followed by ppc1 xenbuilder2 is also quite an old box now. im cleaning up /var/lib/mock on xenbuilder2 now. failed builds dont get cleaned up automatically. we probably should reap them more often.
Buildroots for failed Koji builds do get cleaned up after 4 hours (to give someone time to diagnose the problem if desired). However, xenbuilder2 is also running plague, which never cleans up buildroots as far as I know.
infrastructure@lists.fedoraproject.org
-
Bill Nottingham -
Dennis Gilmore -
Jeremy Katz -
Mike Bonnet -
Nigel Jones