Hi all,
I've not had any builds be scheduled for a couple of hours, is copr stuck / crashed? Thanks.
Richard.
On Thu, 20 Feb 2014 19:10:27 +0000 Richard Hughes hughsient@gmail.com wrote:
Hi all,
I've not had any builds be scheduled for a couple of hours, is copr stuck / crashed? Thanks.
The backend blew up, and I rebooted it...
I'll see if I can unstick processing. ;(
kevin
ok. It's finally processing again I hope. ;)
The problem seemed to be some kind of doom when it was set to 8 workers (so 8 builds at a time). It would launch them all off fine, but then when they returned something would go south and the instance wouldn't be responsive at all. It might be that they all finished at the same time and caused a storm of createrepos or something.
I set it back down to 1 worker (so things will be slow for a bit).
I'll up that to 4 or something tomorrow... and we can see if we can isolate where it is hitting problems.
kevin
On 20 February 2014 21:39, Kevin Fenzi kevin@scrye.com wrote:
ok. It's finally processing again I hope. ;)
Great, thanks.
I'll up that to 4 or something tomorrow... and we can see if we can isolate where it is hitting problems.
I submitted 27 srpms at one time; was this antisocial?
Richard
On Thu, 20 Feb 2014 21:52:04 +0000 Richard Hughes hughsient@gmail.com wrote:
On 20 February 2014 21:39, Kevin Fenzi kevin@scrye.com wrote:
ok. It's finally processing again I hope. ;)
Great, thanks.
I'll up that to 4 or something tomorrow... and we can see if we can isolate where it is hitting problems.
I submitted 27 srpms at one time; was this antisocial?
Naw, it shouldn't be, IMHO. ;)
The backend should cope with that kind of thing...
kevin
On Fri, Feb 21, 2014 at 12:53 AM, Kevin Fenzi kevin@scrye.com wrote:
On Thu, 20 Feb 2014 21:52:04 +0000 Richard Hughes hughsient@gmail.com wrote:
On 20 February 2014 21:39, Kevin Fenzi kevin@scrye.com wrote:
ok. It's finally processing again I hope. ;)
Great, thanks.
I'll up that to 4 or something tomorrow... and we can see if we can isolate where it is hitting problems.
I submitted 27 srpms at one time; was this antisocial?
Naw, it shouldn't be, IMHO. ;)
The backend should cope with that kind of thing...
I think a ring/queue should be written to handle this. I however have not looked into this for sometime. I will bring this up in the channel.
Kind Regards,
kevin
copr-devel mailing list copr-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/copr-devel
On 21 February 2014 09:45, Frankie Onuonga frankie.onuonga@gmail.com wrote:
I think a ring/queue should be written to handle this.
And now http://copr-be.cloud.fedoraproject.org/ seems down too. I'm not having much luck this week.
Richard
On Sat, 22 Feb 2014 10:45:06 +0000 Richard Hughes hughsient@gmail.com wrote:
On 21 February 2014 09:45, Frankie Onuonga frankie.onuonga@gmail.com wrote:
I think a ring/queue should be written to handle this.
And now http://copr-be.cloud.fedoraproject.org/ seems down too. I'm not having much luck this week.
Additionally Mirek is on vacation this week. ;(
Anyhow, the problem is that this backend instance keeps going unresponsive. :(
I've destroyed it and brought it back up.
Hopefully we can track down why it's doing this.
kevin
On 20 February 2014 21:39, Kevin Fenzi kevin@scrye.com wrote:
ok. It's finally processing again I hope. ;)
Down again I think: https://copr.fedoraproject.org/coprs/rhughes/f19-gnome-3-12/builds/
Sorry to be the bearer of bad news....
Richard
On 21 February 2014 09:13, Christopher Meng cickumqt@gmail.com wrote:
Only SSL certificate issue.
Are you sure? I've had builds pending for 8h+...
Richard
On Fri, Feb 21, 2014 at 5:35 PM, Richard Hughes hughsient@gmail.com wrote:
On 21 February 2014 09:13, Christopher Meng cickumqt@gmail.com wrote:
Only SSL certificate issue.
Are you sure? I've had builds pending for 8h+...
Sorry... I misunderstood your comment...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 02/20/2014 10:39 PM, Kevin Fenzi wrote:
ok. It's finally processing again I hope. ;)
The problem seemed to be some kind of doom when it was set to 8 workers (so 8 builds at a time). It would launch them all off fine, but then when they returned something would go south and the instance wouldn't be responsive at all. It might be that they all finished at the same time and caused a storm of createrepos or something.
I set it back down to 1 worker (so things will be slow for a bit).
I'll up that to 4 or something tomorrow... and we can see if we can isolate where it is hitting problems.
I think that the problem was that several threads crashed and there was ~10 instances in Fedora Cloud running without parent so FC may run out of resources. I terminated them and set # workers back to 6. I will try to read logs (if there are some) tomorrow.
I will have to write some maintainaince script which will clean up VM which are not tracked by workers...
Mirek
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 02/20/2014 09:04 PM, Kevin Fenzi wrote:
The backend blew up, and I rebooted it...
Seems to be blew up again. :((
$ ssh root@copr-be.cloud.fedoraproject.org ssh: connect to host copr-be.cloud.fedoraproject.org port 22: No route to host
And even from Fedora Cloud: $ ping 172.16.5.4 PING 172.16.5.4 (172.16.5.4) 56(84) bytes of data. - From 172.16.5.5 icmp_seq=1 Destination Host Unreachable
Kevin, can you get to the console? And check why this happened? And restart it again please?
Mirek
On Sat, 22 Feb 2014 20:13:15 +0100 Miroslav Suchy msuchy@redhat.com wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 02/20/2014 09:04 PM, Kevin Fenzi wrote:
The backend blew up, and I rebooted it...
Seems to be blew up again. :((
$ ssh root@copr-be.cloud.fedoraproject.org ssh: connect to host copr-be.cloud.fedoraproject.org port 22: No route to host
And even from Fedora Cloud: $ ping 172.16.5.4 PING 172.16.5.4 (172.16.5.4) 56(84) bytes of data.
- From 172.16.5.5 icmp_seq=1 Destination Host Unreachable
Kevin, can you get to the console? And check why this happened? And restart it again please?
The console is completely blank.
The logs have a bunch of OOM's from createrepo_c...
Restarted it.
kevin
copr-devel@lists.fedorahosted.org
-
Christopher Meng -
Frankie -
Kevin Fenzi -
Miroslav Suchý -
Richard Hughes