Hi,
I'd like to share a new design proposal which hopefully addresses some of the issues people have when larger jobs fail because of broken hardware.
https://beaker-project.org/dev/proposals/rescheduling-capability.html
Kudos to Bruno Goncalves who brought it initially to my attention.
Is there something the proposal misses? Are there perhaps details which are not correctly flashed out? Do you have questions?
Kind Regards,
----- Original Message -----
From: "Róman Joost" rjoost@redhat.com To: "Beaker Public" beaker-devel@lists.fedorahosted.org Cc: "Bruno Goncalves" bgoncalv@redhat.com Sent: Tuesday, 3 May, 2016 3:31:28 AM Subject: [Beaker-devel] RFC: Design proposal for a re-scheduling capability of recipe sets
Hi,
I'd like to share a new design proposal which hopefully addresses some of the issues people have when larger jobs fail because of broken hardware.
https://beaker-project.org/dev/proposals/rescheduling-capability.htmlKudos to Bruno Goncalves who brought it initially to my attention.
Is there something the proposal misses? Are there perhaps details which are not correctly flashed out? Do you have questions?
Is the only criteria that job aborts? Does it matter how it aborts? If a test uses rhts-abort, is it going to be re-scheduled as well? If a task triggers a panic or kernel hangs, jobs aborts. Will that trigger re-scheduling? Does it matter when it aborts? Is failure to install treated in same way as last task hitting EWD?
Regards, Jan
Kind Regards,
Róman Joost Software Engineer, PnT DevOps - Developer (Brisbane) email: rjoost@redhat.com | tz: UTC+10 | GPG ID: 0xBE2B559D at pgp.mit.edu irc: #beaker #tps-devel
Beaker-devel mailing list beaker-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/beaker-devel@lists.fedorahosted.o...
Excerpts from Jan Stancek's message of 2016-05-03 04:43 -04:00:
Is the only criteria that job aborts? Does it matter how it aborts? If a test uses rhts-abort, is it going to be re-scheduled as well? If a task triggers a panic or kernel hangs, jobs aborts. Will that trigger re-scheduling? Does it matter when it aborts? Is failure to install treated in same way as last task hitting EWD?
Very good points Jan.
I wonder if the re-scheduling should follow the same "suspicious abort" logic we use for broken system detection. That's been changed to and fro a bit lately, but currently an aborted recipe is suspicious if all tasks aborted.
So that would include install failures, and EWD due to never starting the installation -- but would exclude rhts-abort or panics etc (unless the rhts-abort or panic happens in the very first task, but that's normally /distribution/install so it shouldn't happen there).
So that means we're not rescheduling aborted recipes, but suspiciously aborted recipes.
On 05/03/2016 03:31 AM, Róman Joost wrote:
Hi,
I'd like to share a new design proposal which hopefully addresses some of the issues people have when larger jobs fail because of broken hardware.
https://beaker-project.org/dev/proposals/rescheduling-capability.htmlKudos to Bruno Goncalves who brought it initially to my attention.
Is there something the proposal misses? Are there perhaps details which are not correctly flashed out? Do you have questions?
Kind Regards,
I'm glad Beaker is getting rescheduling capabilities!
a) Will the abort and reschedule be atomic? I.e. will it be impossible for beaker-jobwatch[1] to see the aborted recipe before it's cloned?
b)
How to identify re-scheduled recipe sets¶ A UI element will be used to make them distinguishable from regular scheduled recipe sets.
Please make it easily discoverable from the result xml as well. e.g. the aborted recipe could have an attribute rescheduled_as="RS:XYZ"?
c) would it be possible to (optionally?) NAK the aborted recipe?
d) Slightly related: with beaker-jobwatch I struggle to make people investigate the cause of the aborts = if reschedule works they get their results and no longer care that some other recipeset failed to finish. I expect Beaker to have the same issue once it starts "fixing" jobs. Do you have any ideas how to persuade people to investigate these?
/me currently plans to add a phone-home feature to beaker-jobwatch that would log all failed-to-finish recipesets into logstash/ElasticSearch/Kibana, hoping that it will help me spot patterns of failures and report better tickets.
[1] https://wiki.test.redhat.com/BaseOs/Projects/BeakerJobwatch
Beaker-devel mailing list beaker-devel@lists.fedorahosted.org https://lists.fedorahosted.org/admin/lists/beaker-devel@lists.fedorahosted.o...
On Sat, May 21, 2016 at 1:31 AM, Ales Zelinka azelinka@redhat.com wrote:
d) Slightly related: with beaker-jobwatch I struggle to make people investigate the cause of the aborts = if reschedule works they get their results and no longer care that some other recipeset failed to finish. I expect Beaker to have the same issue once it starts "fixing" jobs. Do you have any ideas how to persuade people to investigate these?
/me currently plans to add a phone-home feature to beaker-jobwatch that would log all failed-to-finish recipesets into logstash/ElasticSearch/Kibana, hoping that it will help me spot patterns of failures and report better tickets.
I personally think it would be really cool if Beaker natively supported Elastic-recheck type functionality (given a suitable ELK installation to talk to): http://status.openstack.org/elastic-recheck/
However, the hard part of pursuing that idea wasn't reporting the results, it was getting access to an ELK installation that could plausibly handle the scale of Red Hat's main Beaker installation.
Starting with just beaker-jobwatch managed jobs could help mitigate that by making it possible to calibrate the monitoring capacity needed to make it a standard feature.
Cheers, Nick.
beaker-devel@lists.fedorahosted.org
-
Ales Zelinka -
Dan Callaghan -
Jan Stancek -
Nick Coghlan -
Róman Joost