On 01/23/2014 08:30 AM, Bill Peck wrote:
On 01/22/2014 05:26 PM, Dan Callaghan wrote:
> Excerpts from Jeffrey Burke's message of 2014-01-22 23:11:33 +1000:
>> Nice feature. A couple questions. We have systems the lose console
>> often. What happens with an empty console, does it ignore or assume
>> that the install has failed. or the opposite?
> If the console is missing, empty, or drops out, beaker-watchdog will not
> detect any installation failures so the recipe will continue as normal
> (or eventually the watchdog will get it, if it did fail). That's the
> same way the panic detection works -- it's best effort.
>
>> Also now that the scheduler has detected this failure can it
>> automatically requeue the recipe?
> It could, but that would be a separate feature. I'm not sure if it would
> be worthwhile though. I think almost all cases where the installation
> fails (especially, when it fails in a detectable way) it's due to
> a problem with the recipe: missing a repo, typo in the kickstart,
> requesting filesystem that is too big, etc. Those cases are not likely
> to be solved by resubmitting the recipe.
Except for the dhcp request or failure to download the kickstart. Those
aren't really anaconda errors but transient errors in the infrastructure.
Would this handle those? or is that beyond the scope?
Beyond the scope of this RFE, not necessarily beyond the scope of Beaker :)
The main concern we have around automated resubmission type proposals is
that when there *is* a structural problem with the request, then we need
to figure that out somehow. The two main options are:
- be a little smarter about how we monitor anaconda and treat certain
failures as "looks like a transient infrastructure error, try
automatically restarting the installation".
- try a few times, and if a subsequent attempt works, treat the earlier
ones as a glitch
The two problem with the second approach is that it can hide real
glitches in *Anaconda* (it's not really safe to assume it was an
infrastructure glitch) and that it increases the costs of real failures.
So I think the first approach sounds more promising, but it means even
closer integration with Anaconda, which means an increased risk of new
Anaconda releases breaking Beaker's assumptions. This is one of the
reasons I've been a big proponent of the idea of setting up a Beaker
instance in Fedora QA (which has been progressing thanks to the efforts
of Tim Flink and Alexander Todorov). If we can get to the point where we
can do automated pre-release testing for Anaconda in Fedora QA's Beaker
instance, then that significantly reduces the risks of hitting
unexpected provisioning glitches when new Anaconda versions are released.
We also want to get to a point where most recipes don't need to run
Anaconda at all (by using a prebuilt image in OpenStack instead), but
there's a lot of work still to be done before we get to that level of
sophistication (and I expect the OpenStack integration will bring its
own share of interesting transient infrastructure glitches).
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (
http://beaker-project.org/)