[candlepin] [foreman-dev] Koji is back online

Eric D Helms ericdhelms at gmail.com
Wed Sep 10 21:13:30 UTC 2014


I tested a mash of Katello repos and everything went fine. However,
when I attempted to run the Katello build job I got the following:

http://ci.theforeman.org/job/katello_packaging_build_matrix_rpm_rubygem_katello/4576/label=rpmbuild,os=rhel6/console

152275 build (katello-nightly-rhel6,
ruby193-rubygem-katello-1.5.0-12.201409102050git71a484b.el6.src.rpm):
open (kojibuilder2) -> FAILED: Fault: <Fault 1: 'Traceback (most
recent call last):\n  File
"/usr/lib/python2.6/site-packages/koji/daemon.py", line 1114, in
runTask\n    response = (handler.run(),)\n  File
"/usr/lib/python2.6/site-packages/koji/tasks.py", line 146, in run\n
 return self.handler(*self.params,**self.opts)\n  File
"/usr/sbin/kojid", line 731, in handler\n    h =
self.readSRPMHeader(srpm)\n  File "/usr/sbin/kojid", line 802, in
readSRPMHeader\n    fo = koji.openRemoteFile(relpath, **opts)\n  File
"/usr/lib/python2.6/site-packages/koji/__init__.py", line 1376, in
openRemoteFile\n    fo = open(fn)\nIOError: [Errno 116] Stale file
handle: \'/mnt/koji/work/cli-build/1410382239.066185.bHVVRsri/ruby193-rubygem-katello-1.5.0-12.201409102050git71a484b.el6.src.rpm\'\n'>
  0 free  0 open  0 done  1 failed

Eric

On Wed, Sep 10, 2014 at 4:52 PM, Lukas Zapletal <lzap at redhat.com> wrote:
> Hey,
>
> TL;DR - after another out-of-space incident, Koji is back online
> accepting builds.
>
> I created new 600 GB volume and copied over the content of the /mnt/koji
> plus moved there directories "packages" and "external-repos" from
> /mnt/tmp (ephemeral volume which was filling up).
>
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/xvde1            5.7G  3.0G  2.7G  53% /
> none                  1.8G     0  1.8G   0% /dev/shm
> tmpfs                 4.0G   44M  4.0G   2% /tmp
> /dev/xvdab1           591G  378G  183G  68% /mnt/koji
> /dev/xvdt1             50G   27G   21G  56% /mnt2/koji
> /dev/xvdm1            148G  108G   33G  77% /mnt2/xvdm
> /dev/xvdj1            367G   30G  322G   9% /mnt/tmp
>
> Old volumes xvdt1 and xvdm1 are mounted under /mnt2 and can be removed
> and returned to the EBS pool once we confirm koji is working again.
>
> First, I moved everything under single volume until I realized this was
> deadly slow. For building processes, local storage must be used. This
> was the reason why there was such a mess with symlinks. But due to fact
> that the instance was small flavor, ephemeral volume is small and
> limiting. Fortunately I found "packages" dirs together with
> "external-repos" 'moveable' to EBS volume (Koji is sensitive in
> multi-volume setups beacuse heavy use of hardlinks). Builds can be
> slower in this setup. We should have enough space now (183 G + 322 G).
>
> --
> Later,
>  Lukas #lzap Zapletal
>
> --
> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe at googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



More information about the candlepin mailing list