Adding resource management to beaker
by Bill Peck
Hello Beaker,
One of the jobs we run uses shared storage. For example:
HostA with Driver1
HostB with Driver2
HostC with Driver3
All connected to StorageX.
We can't run the test on more than one host at a time since it would
conflict.
Being able to specify the following in the job xml is the goal:
<recipeSet>
<recipe/> <!-- Require HostA -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostB -->
<resource/> <!-- Require StorageX -->
</recipeSet>
<recipeSet>
<recipe/> <!-- Require HostC -->
<resource/> <!-- Require StorageX -->
</recipeSet>
I know there is a resource type for recipe but this doesn't seem like a
good solution. I did try an experiment where I created StorageX host in
beaker with no power control and scheduled three recipesets like above.
But this hack won't work because a watchdog with a NULL expire time is
set and that means it will sit there forever. Even if it did work,
maybe if we created a dummy power type, we would still end up with an abort.
I don't think it would take much work to support this type of
multi-host/resource requirement.
Right now we have the following model
Object Recipe, which has the following attributes: distro_tree,
resource, rendered_kickstart, watchdog, systems, dyn_systems, tasks,
dyn_tasks, tags, repos, rpms, logs, custom_packages, ks_appends
Object MachineRecipe which inherits from Recipe, and adds a guests
attribute.
Object GuestRecipe which inherits from Recipe.
A ResourceRecipe object wouldn't need most of these attributes, probably
just the following:
resource, systems, and dyn_systems. (no watchdog, just return when
recipeSet completes or aborts)
And resource currently maps to a RecipeResource which can be one of the
following:
SystemResource
VirtResource
GuestResource
I think adding another Object here would make sense, but the term
Resource is overused and its a little confusing right now.
If done correctly we could re-use the filtering code in needproperty,
but I think the only things we could search on for resources would be
the following: Name, lab_controller, and key_values? (storing how these
shared resources are connected would be another option I suppose, but
maybe too complicated for a first implementation?)
I'm sure there are other things to consider, we could support setting up
and tearing down shared resources (almost like a power on/off command).
I want to be mindful of things like that but I don't want to be
paralysed with what if's and never get anything done.
I know you guys have been working hard on the groups features and want
to re-do the scheduler as well. Maybe this would be something for after
the scheduler re-write? I don't see any mention of this on the roadmap
[1] (although the roadmap is quite full already!).
In any event, I'm interested in hearing your ideas and feedback.
Thanks!
1 - http://beaker-project.org/dev/tech-roadmap.html
8 years, 7 months
Simplifying running the inventory task
by Nick Coghlan
Currently, doing a hardware scan on a system requires a command line like the following:
bkr machine-test --inventory --family=RedHatEnterpriseLinux6 --arch=x86_64 --machine=<FQDN>
The "family" part may need to change when using Beaker to manage architectures that RHEL6 doesn't support. However, due to the fact beaker-system-scan currently still depends on smolt for some features, using RHEL7 or Fedora instead isn't ideal if RHEL6 supports the hardware.
In order to eventually add a "Scan this system now" button to the web UI, and, equivalently, a simple "bkr update-inventory <FQDN>" command, I think we may need to move the logic of choosing a preferred distro family to run the inventory task to the main Beaker server.
Specifically, I'm thinking of a mapping of architecture -> distro family + inventory task, with a default distro and inventory task to use for unspecified architectures, together with a dedicated server API *separate* from the normal job submission API, that requested an inventory scan for a particular machine. The task of choosing which distro to use, and which inventory task to run would then be handled by the server.
The purpose of this would be to better enable working with architectures that the default approach can't handle yet - standard supported architectures would use the default of /distribution/inventory-on-RHEL-6 (at least until we manage to remove the dependency on smolt, then we can switch to RHEL7 by default), while experimental architectures could use a different distro, or even a different inventory task.
By making this admin configurable (include the default distro used), we'd also better support instances that don't have RHEL trees loaded at all, but only Fedora and/or CentOS.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
8 years, 9 months
Defining & documenting Beaker's architecture support levels
by Nick Coghlan
Something that came up recently is the question of how we clearly
communicate *how well* Beaker supports different architectures. In
particular, just because we support provisioning something doesn't
necessarily mean we support doing hardware inventory scans on it.
I think the first step in adding something like that to the architecture
guide is to just list out all the features that go into our architecture
support. That's a useful thing to have regardless, and it means we can
potentially go through and create a feature matrix for it (or perhaps
summarise it as a numeric ranking).
On the provisioning side, the items I'm aware of are:
- basic provisioning (i.e. can we boot it at all)
- boot menu (i.e. do we provide a boot menu for manual provisioning)
On the inventory side, the items I'm aware of are:
- basic inventory (i.e. memory, number of CPUs, disk info)
- CPU details (i.e. precise details of the CPU capabilities)
So only x86 and x86_64 would currently get full marks - for everything
else, we don't currently provide full CPU details.
I think this also highlights for me that we *really* want at least a
summary in the architecture guide of what info beaker-system-scan
collects. It doesn't need to be completely detailed in terms of how the
information is reported back to Beaker, but having that list of "what
beaker-system-scan reports back to Beaker" will help explain the purpose
the inventory system serves, and (eventually) what some of the
limitations may be for architectures that aren't fully supported by some
of the scanning tools.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
8 years, 10 months
Offering lshw based inventory scans for new architectures
by Nick Coghlan
As we add new experimental architectures, we're in the situation where
we don't currently have a working inventory scan for those systems,
since newer versions of the operating system that support those
architectures are also missing smolt.
So, here's my question: would it be possible to tweak beaker-system-scan
such that, if smolt is available, we use it in preference to lshw, but
otherwise, we just rely on lshw?
My idea here is that, for existing architectures, we would run the
inventory task on an OS with smolt, to make sure we continue to supply
the same data we always have, until we're sure we have lshw to the point
where it's a suitable replacement.
But for new architectures, supplying whatever lshw can give us today
(along with the rest of the info beaker-system-scan collects) would be
better than nothing, so it makes sense to at least do that, and skip the
parts that depend on smolt.
Does this approach make sense to anyone else? Would it be reasonably
easy to just skip the smolt-based components of the scan when it wasn't
available?
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
8 years, 10 months
Dedicated subcommand for hardware scans?
by Nick Coghlan
I thought I had filed an RFE for this somewhere along the line, but
apparently not...
Currently, hardware scans are requested via the "--inventory" option to
the "machine-test" subcommand. This is a little(!) obscure for a feature
that powers the entire inventory search capability.
What do folks the think of the idea of adding a dedicated
"update-inventory" subcommand such that "bkr update-inventory" was the
equivalent of "bkr machine-test --inventory --ignore-system-status"?
Cheers,
Nick.
--
Nick Coghlan
Red Hat Hosted & Shared Services
Software Engineering & Development, Brisbane
HSS Provisioning Architect
8 years, 10 months
How do I deprecate a workflow?
by Alexander Todorov
Hi guys,
how do I go about deprecating workflow-snake ? There is already a replacement
for it (workflow-installer-test) which uses Jinja2 templates and doesn't need
the additional piece of infrastructure (SNAKE server).
There's a proof of concept that it is usable (public tier #1 test cases for
Fedora).
What are the steps (both in code and in process) to deprecate workflow-snake and
subsequently remove it from beaker-client ?
--
Alex
8 years, 10 months
Proof of concept: Running tests in a Docker container
by Amit Saha
I just posted a draft patch for a proof-of-concept that I got working early this week
for running the tests in a Docker container rather on the test system itself:
http://gerrit.beaker-project.org/#/c/3199/
Considering the ease of running the "restraint" test harness, I have used "restraint" instead
of "beah", Beaker's default test harness.
>From a user's perspective, the only change needed is specifying "contained_harness" as a ksmeta
variable.
It's draft, so comments/suggestions/questions very welcome.
8 years, 11 months
Unplanned downtime for beaker-project.org
by Dan Callaghan
Overnight the filesystem for the VM hosting beaker-project.org filled
up. Right now it looks like nothing bad or unusual happened, it is just
the accumulation of our large yum repos, our many large patchbot
results, and our many nightly builds (it was actually the nightly build
that finally filled it up).
I have pruned some nightlies to free up some space, but I am going to
take the opportunity to perform a thorough audit of the disk space in
use, upgrade to Fedora 20, expand the disk, and move /srv onto
a separate volume so that if it fills up in future it doesn't bring the
whole box grinding to a halt.
Please note that some mails to beaker-project.org were rejected (as far
as I can see they were only spam anyway), there may have been other
glitches as well. Ping me in #beaker if you see any other issues.
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Hosted & Shared Services
Red Hat, Inc.
8 years, 11 months