On Mon, 28 Nov 2011 09:39:28 -0500 (EST)
Kamil Paral <kparal(a)redhat.com> wrote:
I found two crashed on stage from Nov 26:
http://autoqa-stg.fedoraproject.org/results/1972-autotest/virt23.qa/debug/
http://autoqa-stg.fedoraproject.org/results/1990-autotest/virt23.qa/debug/
Can it be a problem in Boji? Can we distinguish whether Boji returned
wrong reply or whether Bodhi returned wrong reply and Boji relayed
that?
Honestly, my money would be on boji causing the problem; it's proxying
the query to bodhi and the error I'm seeing is about an empty response.
The proxy code isn't incredibly robust and if there's a problem, it
wouldn't return anything.
Looking through the logs on autoqa-stg, I see a traceback thrown by
boji. It looks like a problem handling server errors on queries to
bodhi - if an HTTP 500 is returned from bodhi, the proxy code doesn't
know what to do and doesn't return anything. The time doesn't match up
with the errors you linked to but that may not mean much.
My conclusion? More work is needed on boji if we want it to work
better. I have plans to improve the application structure, add better
logging and better testing but as always, it's a matter of making it a
high enough priority to work on. If we're only seeing this every once
in a while, I'm not sure it's too big of a deal for now.
Tim