[candlepin] [foreman-dev] Koji is back online
Eric D Helms
ericdhelms at gmail.com
Wed Sep 10 21:13:30 UTC 2014
I tested a mash of Katello repos and everything went fine. However,
when I attempted to run the Katello build job I got the following:
http://ci.theforeman.org/job/katello_packaging_build_matrix_rpm_rubygem_katello/4576/label=rpmbuild,os=rhel6/console
152275 build (katello-nightly-rhel6,
ruby193-rubygem-katello-1.5.0-12.201409102050git71a484b.el6.src.rpm):
open (kojibuilder2) -> FAILED: Fault: <Fault 1: 'Traceback (most
recent call last):\n File
"/usr/lib/python2.6/site-packages/koji/daemon.py", line 1114, in
runTask\n response = (handler.run(),)\n File
"/usr/lib/python2.6/site-packages/koji/tasks.py", line 146, in run\n
return self.handler(*self.params,**self.opts)\n File
"/usr/sbin/kojid", line 731, in handler\n h =
self.readSRPMHeader(srpm)\n File "/usr/sbin/kojid", line 802, in
readSRPMHeader\n fo = koji.openRemoteFile(relpath, **opts)\n File
"/usr/lib/python2.6/site-packages/koji/__init__.py", line 1376, in
openRemoteFile\n fo = open(fn)\nIOError: [Errno 116] Stale file
handle: \'/mnt/koji/work/cli-build/1410382239.066185.bHVVRsri/ruby193-rubygem-katello-1.5.0-12.201409102050git71a484b.el6.src.rpm\'\n'>
0 free 0 open 0 done 1 failed
Eric
On Wed, Sep 10, 2014 at 4:52 PM, Lukas Zapletal <lzap at redhat.com> wrote:
> Hey,
>
> TL;DR - after another out-of-space incident, Koji is back online
> accepting builds.
>
> I created new 600 GB volume and copied over the content of the /mnt/koji
> plus moved there directories "packages" and "external-repos" from
> /mnt/tmp (ephemeral volume which was filling up).
>
> Filesystem Size Used Avail Use% Mounted on
> /dev/xvde1 5.7G 3.0G 2.7G 53% /
> none 1.8G 0 1.8G 0% /dev/shm
> tmpfs 4.0G 44M 4.0G 2% /tmp
> /dev/xvdab1 591G 378G 183G 68% /mnt/koji
> /dev/xvdt1 50G 27G 21G 56% /mnt2/koji
> /dev/xvdm1 148G 108G 33G 77% /mnt2/xvdm
> /dev/xvdj1 367G 30G 322G 9% /mnt/tmp
>
> Old volumes xvdt1 and xvdm1 are mounted under /mnt2 and can be removed
> and returned to the EBS pool once we confirm koji is working again.
>
> First, I moved everything under single volume until I realized this was
> deadly slow. For building processes, local storage must be used. This
> was the reason why there was such a mess with symlinks. But due to fact
> that the instance was small flavor, ephemeral volume is small and
> limiting. Fortunately I found "packages" dirs together with
> "external-repos" 'moveable' to EBS volume (Koji is sensitive in
> multi-volume setups beacuse heavy use of hardlinks). Builds can be
> slower in this setup. We should have enough space now (183 G + 322 G).
>
> --
> Later,
> Lukas #lzap Zapletal
>
> --
> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
More information about the candlepin
mailing list