[Beaker-devel] Implicit retries considered useful?

Dan Callaghan dcallagh at redhat.com
Wed Jul 10 05:04:18 UTC 2013


Excerpts from Nick Coghlan's message of 2013-06-26 19:04:08 +1000:
> To help narrow down a problem that sometimes occurs with lab controller
> <-> main server communications, Beaker 0.13.2 is currently scheduled to
> completely remove the implicit retries for the affected XML-RPC calls
> (see https://bugzilla.redhat.com/show_bug.cgi?id=974352).
> 
> However, it occurs to me that there's a potentially big downside to
> doing this: currently, we can usually restart the main web service (e.g.
> for a maintenance update that doesn't include any database changes)
> without significant ill effects, because it will just trip the automatic
> retries in the lab controllers for any in progress calls and those calls
> will generally succeed on the second attempt.
> 
> By turning the implicit retries off *completely* for the lab controller
> daemons (rather then just reducing the number of attempts and logging
> each failed attempt properly), any calls back to the server when the
> server shuts down will actually *fail* completely.
> 
> So while our initial assessment was that the implicit retries were doing
> more harm than good, I think this may be a case where we actually need
> them. If I'm right, then 0.13.2 isn't shippable in its current form (we
> need to either revert that change to the lab controllers, or update it
> to still retry at least once, and log the first failure properly).

So I'm a bit late to this particular party due to being on PTO. And 
I see that we did indeed go back to the implicit retries for 0.13.2.

But I think ultimately removing them *is* the right thing to do, for all 
the same reasons we originally wanted to: XML-RPC gives no way to 
indicate if a call is idempotent, and many calls are not, and the retry 
can end up potentially repeating an already completed call and generally 
making a mess of things.

The right thing to do is to retry at the application level, when it 
makes sense -- or more likely, just skip the call and try again later. 
Apart from beaker-proxy all the LC daemons are polling loops, so they 
should follow the same model as beaker-provision already does:

* authenticate with the server once before daemonizing, if this fails 
  then refuse to start (under the assumption the config is wrong)
* enter the main polling loop, if a polling call fails just log it and 
  wait for the next iteration
* when handling commands, if a call fails just log it and bail out from 
  handling that particular command

That way the daemon will not fall over if the server goes away 
temporarily for any reason (e.g. for upgrade) while they are running.

I think we are all already on the same page with this but I just wanted 
to note it in this thread.

-- 
Dan Callaghan <dcallagh at redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <https://lists.fedorahosted.org/pipermail/beaker-devel/attachments/20130710/c5f82691/attachment.sig>


More information about the Beaker-devel mailing list