Adding resource management to beaker
by Bill Peck
Hello Beaker,
One of the jobs we run uses shared storage. For example:
HostA with Driver1
HostB with Driver2
HostC with Driver3
All connected to StorageX.
We can't run the test on more than one host at a time since it would
conflict.
Being able to specify the following in the job xml is the goal:
<recipeSet>
<recipe/> <!-- Require HostA -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostB -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostC -->
<resource/> <!-- Require StorageX -->
</recipeSet>
I know there is a resource type for recipe but this doesn't seem like a
good solution. I did try an experiment where I created StorageX host in
beaker with no power control and scheduled three recipesets like above.
But this hack won't work because a watchdog with a NULL expire time is
set and that means it will sit there forever. Even if it did work,
maybe if we created a dummy power type, we would still end up with an abort.
I don't think it would take much work to support this type of
multi-host/resource requirement.
Right now we have the following model
Object Recipe, which has the following attributes: distro_tree,
resource, rendered_kickstart, watchdog, systems, dyn_systems, tasks,
dyn_tasks, tags, repos, rpms, logs, custom_packages, ks_appends
Object MachineRecipe which inherits from Recipe, and adds a guests
attribute.
Object GuestRecipe which inherits from Recipe.
A ResourceRecipe object wouldn't need most of these attributes, probably
just the following:
resource, systems, and dyn_systems. (no watchdog, just return when
recipeSet completes or aborts)
And resource currently maps to a RecipeResource which can be one of the
following:
SystemResource
VirtResource
GuestResource
I think adding another Object here would make sense, but the term
Resource is overused and its a little confusing right now.
If done correctly we could re-use the filtering code in needproperty,
but I think the only things we could search on for resources would be
the following: Name, lab_controller, and key_values? (storing how these
shared resources are connected would be another option I suppose, but
maybe too complicated for a first implementation?)
I'm sure there are other things to consider, we could support setting up
and tearing down shared resources (almost like a power on/off command).
I want to be mindful of things like that but I don't want to be
paralysed with what if's and never get anything done.
I know you guys have been working hard on the groups features and want
to re-do the scheduler as well. Maybe this would be something for after
the scheduler re-write? I don't see any mention of this on the roadmap
[1] (although the roadmap is quite full already!).
In any event, I'm interested in hearing your ideas and feedback.
Thanks!
1 - http://beaker-project.org/dev/tech-roadmap.html
9 years, 1 month
External tasks proposal up on beaker-project.org
by Nick Coghlan
I decided it made more sense to publish Bill's external tasks library on
beaker-project.org and start a thread here, rather than continue to
conduct the high level design discussions in the Gerrit review.
Firstly, I like the idea of "fetch" and "rpm" nodes in the task XML. The
proposal should be more explicit about the fact that the RPM node is
only present in the recipe XML when it is retrieved through the stable
harness API, and not in other cases (such as when cloning a job).
I *don't* like the way the proposal refers to *removing* the task
library. The task library isn't going anywhere, we're just providing a
way to bypass it when doing so makes sense. The task library is still an
incredibly convenient way to publish utilities like the standard tasks
(http://beaker-project.org/docs/user-guide/beaker-provided-tasks.html),
and potentially future higher level utilities like properly configured
KDC, LDAP and qpid servers.
Finally, I'd prefer it if the proposal was backed by a core
"/distribution/gitfetch" task, together with the ability to flag a task
in the task library as "external". When a task was flagged as external,
then Beaker would *require* that a "fetch" element exist, and would
display that URL when appropriate. Given the UI issues in the Beaker
0.15 upgrade, I'm concerned about breaking the fundamental assumption of
every task being associated with an entry in the task library. By using
an "external" flag and an implied task name (instead of leaving the task
library reference NULL), it changes the failure mode from failing to
handle an unexpected NULL or None value to display the task name as
usual rather than the desired URL details.
Regards,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
9 years, 10 months
Beaker 0.15.3 released!
by Nick Coghlan
On behalf of the Beaker development team, I'm pleased to announce that
Beaker v0.15.3 is now available from beaker-project.org [1].
In addition to releasing a new version of Beaker itself, we have also
released new versions of the default beah test harness, the rhts test
environment and development library and the standard reservesys and
virt/install tasks.
Some of the most notable changes for all Beaker users include:
* The new beah release includes preliminary support for IPv6
testing [2].
* Beaker now monitors the console log (if available) for failures
during Anaconda installation and terminates the recipe promptly
with an appropriate message rather than waiting for the external
watchdog to time out.
* The autopartitioning scheme used on newer versions of Anaconda is
now configurable in the recipe XML.
* The new "bkr system-status" subcommand provides remote access to
various aspects of the system state that were previously only
available in the main web UI.
* The CSS for the system page has been adjusted to be more readable
in the new Bootstrap based theme. Longer term, the plan is to
redesign the system page to properly account for the additional
features it has accumulated over time [3].
For Beaker administrators, additional notable changes include:
* The new ``beaker-create-kickstart`` command line utility makes
it easier to create and debug kickstart templates and snippets.
* The system CSV export and import feature now support bulk
renaming of systems.
A more detailed description of these and other changes can be found in
the updated Beaker 0.15 release notes [4].
The detailed list of all changes made since Beaker v0.15.2 is also
available [5].
Regards,
Nick.
[1] http://beaker-project.org/releases/
[2]
http://beah.readthedocs.org/en/latest/admin.html#using-beah-for-ipv6-testing
[3]
http://beaker-project.org/dev/proposals/system-page-improvements.html#sys...
[4] http://beaker-project.org/docs/whats-new/release-0.15.html#beaker-0-15-3
[5] http://beaker-project.org/releases/beaker-0.15.3-ChangeLog
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
9 years, 10 months
Compatibility issue with the new rhts default task RPM naming
by Nick Coghlan
With the current master branch in RHTS, the default behaviour of "make
rpm" and "make bkradd" is now to leave out the repo name.
However, due to the way the per-recipe task library snapshotting
currently works (as described in
https://bugzilla.redhat.com/show_bug.cgi?id=1055815), I believe that's a
problematic change until we fix
https://bugzilla.redhat.com/show_bug.cgi?id=1055821 in order to handle
task RPM *renames* in addition to outright deletion.
If we release RHTS in its current state, people may generate RPMs with
the new names, upload them, and then find
scheduled-but-not-yet-executing jobs aborting as they're told to try to
find the *new* name in the *old* snapshot, which isn't going to work.
The trick of providing the old package name in the new RPM that we
thought handled this doesn't actually help, because it gets the problem
backwards.
This is why we ended up having to add the RHTS_RPM_NAME to override the
names of our relocated tasks to match the names they had in the old repo.
So, I think we need to at least partially revert that particular change
and keep using the legacy repo-specific names until BZ#1055821 has been
implemented. For our use case (handling the relocation from the main
beaker repo to the separate beaker-core-tasks repo without tripping over
the renaming bug), the RHTS_RPM_NAME override does the trick.
Regards,
Nick.
P.S. Yes, http://beaker-project.org/dev/proposals/external-tasks.html is
*definitely* going to be the focus of 0.16 :)
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
9 years, 10 months
Faster detection of failed Anaconda installs merged
by Nick Coghlan
Another one from the "notable merges" files is Dan Callaghan's changes
to add the ability to automatically detect installer failures from the
console log output and abort the recipe immediately rather than waiting
for the external watch dog to time out:
http://gerrit.beaker-project.org/#/c/2696/
This feature will be part of the upcoming 0.15.3 release, and should
result in faster feedback cycles for anyone working on installer issues,
as well as reducing the time consumed on systems when attempting to test
against trees that contain a broken installer version.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
9 years, 10 months
IPv6 support merged for the default test harness (beah)
by Nick Coghlan
Amit Saha achieved a significant milestone for Beaker this morning, by
finally merging IPv6 support into the main line of development for the
default harness, beah: http://gerrit.beaker-project.org/#/c/2694/
This has been a long time coming, and has involved quite a bit of
investigation into the quirks of IPv6 support across different versions
of Fedora, RHEL and Twisted (as well as determining the additional
requirements needed in a lab to handle IPv6-only testing).
There is still some more testing and documentation work to go, but this
should make its way into a new harness release within the next couple of
weeks :)
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
9 years, 10 months
Some upcoming road map changes
by Nick Coghlan
At the moment, if you look at
http://beaker-project.org/dev/proposals/handling-large-installations.html it
says that Beaker 0.16 will focus on various workflow changes in the
system page.
Dan Callaghan has been working on those changes internally, and while
we're really happy with how they have worked out, we also realised they
need to properly communicated externally before we include them in a
release. (If you want to try them out, Dan's patch series targeting the
new-system-page feature branch has all the details. This is the
currently the final patch in that chain:
http://gerrit.beaker-project.org/#/c/2687/)
Accordingly, we're going to be adjusting the major user facing features
included in the next few releases.
Short version:
* Beaker 0.15.3: CLI updates, system page styling update
* Beaker 0.16: git based task execution
* Beaker 0.17 (probably): system page updates
Timeline wise, we currently expect 0.15.3 to be available in late
January, 0.16 in mid February, and 0.17 some time after that.
Long version:
* the CLI additions currently mentioned under Beaker 0.16 that weren't
already included in Beaker 0.15.2 have all been included in the upcoming
Beaker 0.15.3 maintenance release instead. Beaker 0.15.3 will also
include some styling improvements to the system page, as that page
really didn't fare well in the original transition to the new Bootstrap
based web UI styling.
* rather than the major system page update, the Beaker 0.16 release will
instead focus on incorporating Bill Peck's proposal (and patch) for
execution of tasks directly from git
(http://beaker-project.org/dev/proposals/external-tasks.html). This
should resolve most of the major versioning issues currently associated
with the RPM task library, since users will be free to define their own
versioning based on git branches, repo names, etc. The task library
itself won't be going anywhere, however - in addition to the obvious
backwards compatibility requirements, it's also a convenient way to
distribute the standard tasks that are part of Beaker's normal operation
like reservesys, uploading Anaconda install logs, configuring and
starting guest recipes and running the inventory scripts.
* we will also be preparing a design proposal that describes in detail
the changes being made to the system page, along with the reasons for
each of them. Some of the current workflows (especially in 0.14 and
earlier) are quite error prone and confusing, and the only reason for
that is that the old web UI infrastructure made it difficult to improve
them. With the new web UI stack introduced in 0.15, we now have
substantially better development tools at our disposal, and are bringing
that to bear to make the system page more intuitive and to remove the
duplication between some aspects of the provisioning tab and the system
reservation workflow.
The aim will then be to land the updated system page in Beaker 0.17,
partly because we don't want to maintain that feature branch outside the
main line of development for too long, but mostly because we think it
really does represent a major enhancement to Beaker's usability.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
Testing Solutions Team Lead
Beaker Development Lead (http://beaker-project.org/)
9 years, 10 months