To help narrow down a problem that sometimes occurs with lab controller
<-> main server communications, Beaker 0.13.2 is currently scheduled to
completely remove the implicit retries for the affected XML-RPC calls
(see
https://bugzilla.redhat.com/show_bug.cgi?id=974352).
However, it occurs to me that there's a potentially big downside to
doing this: currently, we can usually restart the main web service (e.g.
for a maintenance update that doesn't include any database changes)
without significant ill effects, because it will just trip the automatic
retries in the lab controllers for any in progress calls and those calls
will generally succeed on the second attempt.
By turning the implicit retries off *completely* for the lab controller
daemons (rather then just reducing the number of attempts and logging
each failed attempt properly), any calls back to the server when the
server shuts down will actually *fail* completely.
So while our initial assessment was that the implicit retries were doing
more harm than good, I think this may be a case where we actually need
them. If I'm right, then 0.13.2 isn't shippable in its current form (we
need to either revert that change to the lab controllers, or update it
to still retry at least once, and log the first failure properly).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (
http://beaker-project.org/)