Excerpts from Nick Coghlan's message of 2013-06-26 19:04:08 +1000:
To help narrow down a problem that sometimes occurs with lab
<-> main server communications, Beaker 0.13.2 is currently scheduled to
completely remove the implicit retries for the affected XML-RPC calls
However, it occurs to me that there's a potentially big downside to
doing this: currently, we can usually restart the main web service (e.g.
for a maintenance update that doesn't include any database changes)
without significant ill effects, because it will just trip the automatic
retries in the lab controllers for any in progress calls and those calls
will generally succeed on the second attempt.
By turning the implicit retries off *completely* for the lab controller
daemons (rather then just reducing the number of attempts and logging
each failed attempt properly), any calls back to the server when the
server shuts down will actually *fail* completely.
So while our initial assessment was that the implicit retries were doing
more harm than good, I think this may be a case where we actually need
them. If I'm right, then 0.13.2 isn't shippable in its current form (we
need to either revert that change to the lab controllers, or update it
to still retry at least once, and log the first failure properly).
So I'm a bit late to this particular party due to being on PTO. And
I see that we did indeed go back to the implicit retries for 0.13.2.
But I think ultimately removing them *is* the right thing to do, for all
the same reasons we originally wanted to: XML-RPC gives no way to
indicate if a call is idempotent, and many calls are not, and the retry
can end up potentially repeating an already completed call and generally
making a mess of things.
The right thing to do is to retry at the application level, when it
makes sense -- or more likely, just skip the call and try again later.
Apart from beaker-proxy all the LC daemons are polling loops, so they
should follow the same model as beaker-provision already does:
* authenticate with the server once before daemonizing, if this fails
then refuse to start (under the assumption the config is wrong)
* enter the main polling loop, if a polling call fails just log it and
wait for the next iteration
* when handling commands, if a call fails just log it and bail out from
handling that particular command
That way the daemon will not fall over if the server goes away
temporarily for any reason (e.g. for upgrade) while they are running.
I think we are all already on the same page with this but I just wanted
to note it in this thread.
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.