Beaker-devel July 2014

beaker-devel@lists.fedorahosted.org

4 participants
10 discussions

Adding resource management to beaker
by Bill Peck 21 Oct '14

21 Oct '14

Hello Beaker, One of the jobs we run uses shared storage. For example: HostA with Driver1 HostB with Driver2 HostC with Driver3 All connected to StorageX. We can't run the test on more than one host at a time since it would conflict. Being able to specify the following in the job xml is the goal: <recipeSet> <recipe/>  <resource/>  </recipeSet> <recipeSet> <recipe/>  <resource/>  </recipeSet> <recipeSet> <recipe/>  <resource/>  </recipeSet> I know there is a resource type for recipe but this doesn't seem like a good solution. I did try an experiment where I created StorageX host in beaker with no power control and scheduled three recipesets like above. But this hack won't work because a watchdog with a NULL expire time is set and that means it will sit there forever. Even if it did work, maybe if we created a dummy power type, we would still end up with an abort. I don't think it would take much work to support this type of multi-host/resource requirement. Right now we have the following model Object Recipe, which has the following attributes: distro_tree, resource, rendered_kickstart, watchdog, systems, dyn_systems, tasks, dyn_tasks, tags, repos, rpms, logs, custom_packages, ks_appends Object MachineRecipe which inherits from Recipe, and adds a guests attribute. Object GuestRecipe which inherits from Recipe. A ResourceRecipe object wouldn't need most of these attributes, probably just the following: resource, systems, and dyn_systems. (no watchdog, just return when recipeSet completes or aborts) And resource currently maps to a RecipeResource which can be one of the following: SystemResource VirtResource GuestResource I think adding another Object here would make sense, but the term Resource is overused and its a little confusing right now. If done correctly we could re-use the filtering code in needproperty, but I think the only things we could search on for resources would be the following: Name, lab_controller, and key_values? (storing how these shared resources are connected would be another option I suppose, but maybe too complicated for a first implementation?) I'm sure there are other things to consider, we could support setting up and tearing down shared resources (almost like a power on/off command). I want to be mindful of things like that but I don't want to be paralysed with what if's and never get anything done. I know you guys have been working hard on the groups features and want to re-do the scheduler as well. Maybe this would be something for after the scheduler re-write? I don't see any mention of this on the roadmap [1] (although the roadmap is quite full already!). In any event, I'm interested in hearing your ideas and feedback. Thanks! 1 - http://beaker-project.org/dev/tech-roadmap.html

2 2

Simplifying running the inventory task
by Nick Coghlan 12 Aug '14

12 Aug '14

Currently, doing a hardware scan on a system requires a command line like the following: bkr machine-test --inventory --family=RedHatEnterpriseLinux6 --arch=x86_64 --machine=<FQDN> The "family" part may need to change when using Beaker to manage architectures that RHEL6 doesn't support. However, due to the fact beaker-system-scan currently still depends on smolt for some features, using RHEL7 or Fedora instead isn't ideal if RHEL6 supports the hardware. In order to eventually add a "Scan this system now" button to the web UI, and, equivalently, a simple "bkr update-inventory <FQDN>" command, I think we may need to move the logic of choosing a preferred distro family to run the inventory task to the main Beaker server. Specifically, I'm thinking of a mapping of architecture -> distro family + inventory task, with a default distro and inventory task to use for unspecified architectures, together with a dedicated server API *separate* from the normal job submission API, that requested an inventory scan for a particular machine. The task of choosing which distro to use, and which inventory task to run would then be handled by the server. The purpose of this would be to better enable working with architectures that the default approach can't handle yet - standard supported architectures would use the default of /distribution/inventory-on-RHEL-6 (at least until we manage to remove the dependency on smolt, then we can switch to RHEL7 by default), while experimental architectures could use a different distro, or even a different inventory task. By making this admin configurable (include the default distro used), we'd also better support instances that don't have RHEL trees loaded at all, but only Fedora and/or CentOS. Cheers, Nick. -- Nick Coghlan Red Hat Hosted & Shared Services Software Engineering & Development, Brisbane HSS Provisioning Architect

2 3

Defining & documenting Beaker's architecture support levels
by Nick Coghlan 28 Jul '14

28 Jul '14

Something that came up recently is the question of how we clearly communicate *how well* Beaker supports different architectures. In particular, just because we support provisioning something doesn't necessarily mean we support doing hardware inventory scans on it. I think the first step in adding something like that to the architecture guide is to just list out all the features that go into our architecture support. That's a useful thing to have regardless, and it means we can potentially go through and create a feature matrix for it (or perhaps summarise it as a numeric ranking). On the provisioning side, the items I'm aware of are: - basic provisioning (i.e. can we boot it at all) - boot menu (i.e. do we provide a boot menu for manual provisioning) On the inventory side, the items I'm aware of are: - basic inventory (i.e. memory, number of CPUs, disk info) - CPU details (i.e. precise details of the CPU capabilities) So only x86 and x86_64 would currently get full marks - for everything else, we don't currently provide full CPU details. I think this also highlights for me that we *really* want at least a summary in the architecture guide of what info beaker-system-scan collects. It doesn't need to be completely detailed in terms of how the information is reported back to Beaker, but having that list of "what beaker-system-scan reports back to Beaker" will help explain the purpose the inventory system serves, and (eventually) what some of the limitations may be for architectures that aren't fully supported by some of the scanning tools. Cheers, Nick. -- Nick Coghlan Red Hat Hosted & Shared Services Software Engineering & Development, Brisbane HSS Provisioning Architect

2 1

Offering lshw based inventory scans for new architectures
by Nick Coghlan 28 Jul '14

28 Jul '14

As we add new experimental architectures, we're in the situation where we don't currently have a working inventory scan for those systems, since newer versions of the operating system that support those architectures are also missing smolt. So, here's my question: would it be possible to tweak beaker-system-scan such that, if smolt is available, we use it in preference to lshw, but otherwise, we just rely on lshw? My idea here is that, for existing architectures, we would run the inventory task on an OS with smolt, to make sure we continue to supply the same data we always have, until we're sure we have lshw to the point where it's a suitable replacement. But for new architectures, supplying whatever lshw can give us today (along with the rest of the info beaker-system-scan collects) would be better than nothing, so it makes sense to at least do that, and skip the parts that depend on smolt. Does this approach make sense to anyone else? Would it be reasonably easy to just skip the smolt-based components of the scan when it wasn't available? Cheers, Nick. -- Nick Coghlan Red Hat Hosted & Shared Services Software Engineering & Development, Brisbane HSS Provisioning Architect

3 2

Dedicated subcommand for hardware scans?
by Nick Coghlan 21 Jul '14

21 Jul '14

I thought I had filed an RFE for this somewhere along the line, but apparently not... Currently, hardware scans are requested via the "--inventory" option to the "machine-test" subcommand. This is a little(!) obscure for a feature that powers the entire inventory search capability. What do folks the think of the idea of adding a dedicated "update-inventory" subcommand such that "bkr update-inventory" was the equivalent of "bkr machine-test --inventory --ignore-system-status"? Cheers, Nick. -- Nick Coghlan Red Hat Hosted & Shared Services Software Engineering & Development, Brisbane HSS Provisioning Architect

2 2

How do I deprecate a workflow?
by Alexander Todorov 18 Jul '14

18 Jul '14

Hi guys, how do I go about deprecating workflow-snake ? There is already a replacement for it (workflow-installer-test) which uses Jinja2 templates and doesn't need the additional piece of infrastructure (SNAKE server). There's a proof of concept that it is usable (public tier #1 test cases for Fedora). What are the steps (both in code and in process) to deprecate workflow-snake and subsequently remove it from beaker-client ? -- Alex

3 3

Beaker 0.17.1 released
by Dan Callaghan 18 Jul '14

18 Jul '14

The Beaker 0.17.1 bug fix release is now available from beaker-project.org [1]. This release fixes a potential data leak, where system power settings were exposed to all authenticated users through CSV export (even if they did not have permission to edit the system) [2]. A number of other bug fixes are also included in this release, and new versions of the Beah test harness and /distribution/virt/install task have been published. As always, please refer to the updated release notes for a complete list [3]. The detailed list of all changes made since Beaker 0.17.0 is also available [4]. [1] https://beaker-project.org/releases/ [2] https://bugzilla.redhat.com/show_bug.cgi?id=1116722 [3] https://beaker-project.org/docs/whats-new/release-0.17.html#beaker-0-17-1 [4] https://git.beaker-project.org/cgit/beaker/log/?qt=range&q=beaker-0.17.0..b… -- Dan Callaghan <dcallagh(a)redhat.com> Software Engineer, Hosted & Shared Services Red Hat, Inc.

1 0

Proof of concept: Running tests in a Docker container
by Amit Saha 11 Jul '14

11 Jul '14

I just posted a draft patch for a proof-of-concept that I got working early this week for running the tests in a Docker container rather on the test system itself: http://gerrit.beaker-project.org/#/c/3199/ Considering the ease of running the "restraint" test harness, I have used "restraint" instead of "beah", Beaker's default test harness. >From a user's perspective, the only change needed is specifying "contained_harness" as a ksmeta variable. It's draft, so comments/suggestions/questions very welcome.

1 0

Pre-defined host filters for workflow commands
by Amit Saha 10 Jul '14

10 Jul '14

A new feature for the beaker client - "pre-defined host filters" was recently merged into develop and will be released when beaker-0.18 is released. [1] It allows you to specify pre-defined host filters to insert custom host selection criteria instead of having to type in the XML everytime when using the workflow commands. Further, it allows you to define your own host filters or override an existing one [2]. [1] https://beaker-project.org/docs-develop/whats-new/next/workflow-preset-host… [2] https://beaker-project.org/docs-develop/man/bkr.html#cmdoption-bkr--host-fi… Best, Amit.

2 1

Unplanned downtime for beaker-project.org
by Dan Callaghan 04 Jul '14

04 Jul '14

Overnight the filesystem for the VM hosting beaker-project.org filled up. Right now it looks like nothing bad or unusual happened, it is just the accumulation of our large yum repos, our many large patchbot results, and our many nightly builds (it was actually the nightly build that finally filled it up). I have pruned some nightlies to free up some space, but I am going to take the opportunity to perform a thorough audit of the disk space in use, upgrade to Fedora 20, expand the disk, and move /srv onto a separate volume so that if it fills up in future it doesn't bring the whole box grinding to a halt. Please note that some mails to beaker-project.org were rejected (as far as I can see they were only spam anyway), there may have been other glitches as well. Ping me in #beaker if you see any other issues. -- Dan Callaghan <dcallagh(a)redhat.com> Software Engineer, Hosted & Shared Services Red Hat, Inc.

1 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Beaker-devel July 2014