New Beaker API quirks
by Don Zickus
Hi Dan,
I have been enjoying the ease to code against the new Beaker API. The
GET, POST, PUSH stuff keeps things really simple. But during my testing I
noticed a few quirks that Bill P. suggested I email you about for your
opinion.
- When I POST/PUT to a url, if the url has an extra '/' in the path, like
/recipes/(recipe_id)/tasks/(task_id)//status
the command gets rejected. Not sure if that is an http thing or a server
thing. Not to much of a problem, just forces me to clean up my code
(which is good!). Just thought I would mention it.
- when I PUT a zero length file to the server, it fails with a 400 code.
Now I should be filtering those (and will now). But it did take me an
hour before I realized that of my list of files, the rejected ones were of
zero length. Again could be on purpose, not sure.
- Because the server does not have a mechanism to 'push' and abort the
client (other than power off), the client could still be running and
trying to pushing results back to the server.
a POST of a task_result gave me an 'Internal Server Error 500'. Will
that be the expected response? I can trap for that and abort the client.
PUT of logs is still allowed even though the task aborted on the server
side. Bill thought that should be fixed. Though I am not sure what the
expectations are.
That's it so far.
Thanks,
Don
10 years, 10 months
Revised Enhanced User Group proposal on Gerrit
by Nick Coghlan
Rather than updating the design proposal directly on the site, I decided
I wanted some feedback on the current draft first:
http://gerrit.beaker-project.org/#/c/1908/1/dev/proposals/enhanced-user-g...
Key changes:
1. The "job modification" flag on group members is gone, replaced with
the separate concept of a "submission delegate", who can submit jobs on
behalf of the group, but cannot modify other group jobs
2. Unlike the previous design, submission delegates retain the ability
to modify the jobs they submit. The design we discussed in the other
thread (where you could change the actual submitting *user*) has been
listed as a deferred feature (because it would require quite a bit more
additional UI design work)
Another couple of points came up while I was writing it:
- should we allow *completed* single-user jobs to be converted to group
jobs? We originally didn't allow this due to the problem with getting
SSH keys installed, but that doesn't apply if the job is already
completed, and it may be necessary if we want to eventually remove the
existing not-quite-group-jobs behaviour (I don't think we can remove
that in 1.0 - we need to allow at least a release for people to migrate
to the new tools before we take the old mechanism away).
- LDAP derived groups can't be self-administered in the current design,
but should we at least allow Beaker administrators to nominate
submission delegates for them?
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 10 months
Clarification request for Enhanced User Groups
by Nick Coghlan
<moving to the public dev list, no reason for this to be in private email>
On 04/29/2013 08:01 PM, Raymond Mancy wrote:
> Just a couple of questions regarding this following sentence:
>
> All members of the group with job modification permissions will be able to ack/nack, change priority,
> edit whiteboard, and change retention tag.
>
> 'All members of the group with job modification permissions', does this actually translate to 'All members
> of the job's group owner'. I'm assuming it is, I was just momentarily confused by it.
No, it's not enough to be a member of the owning group, your membership
in that group also has to be flagged as "can modify jobs".
See the first part of the proposal (before the "Group ownership" section):
==========
All members of a group are able to submit jobs on behalf of that group.
Group members may also be granted the following additional permissions
on a group:
* group ownership
* group job modification
==========
This means there are two independent permissions on groups, above and
beyond the "can submit jobs on behalf of the group" capability:
1. Group ownership
- can add/remove members
- can change owner status of members
- can change job modification permissions of members
2. Job modification
- can modify jobs owned by the group after they are submitted
The intent of this separation is to allow an automated process to be
given the ability to submit jobs on behalf of the group, *without*
granting the automated tool's account the ability to modify those jobs
after creation.
While technically a group owner may not necessarily have job
modification permissions, they can grant themselves those permissions at
any time, so it's not a real restriction (from the code's point of view,
it means we can ignore the group ownership flag when checking for job
modification permissions).
The default settings when adding a new member to a group are:
- Group Owner?: False
- Can Modify Jobs?: True
> Also, is there a reason why we can't afford the group owner the exact same permissions as the
> job submitter (specifically Delete, which is not mentioned)? This would probably make the docs a bit
> more maintainable (as in '...with job modification permissions equal to that of the job submitter' rather than
> having to update it every time a new job action is created). This would also make the code simpler.
It's meant to be full access - if I missed any current capabilities,
it's just a mistake in the design.
However, note that part of the redesign is that the job submitter may
*not* have permissions to make changes after submission, if they're
submitting the job on behalf of a group and they don't have the job
modification flag for that group.
Cheers,
Nick.
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 10 months
Beaker 0.12 released
by Nick Coghlan
Hello everyone,
On behalf of the Beaker developers, I am pleased to announce that Beaker
0.12 has been released. More than 50 bugs (including new features) were
fixed in this release.
Some of the new features introduced in this release are:
- Provisional support for alternative harnesses.
- Live role environment variables.
- New system loan interface and comment fields.
- Disk information in inventory.
A number of other notable changes (some of them potentially backwards
incompatible) were made. These include:
- Asynchronous updates on job status (to eliminate several race conditions).
- Task names limited to 255 characters (to allow indexing and hence
faster searching).
- Kickstart sections are properly terminated for RHEL 6 (to ensure
correct behaviour when making additional repos available during
installation).
Pre-built packages and source archives are available for download from
here [1].
To read the release notes in full please go to [2]. The project
documentation is available at [3].
Due to some regressions [5] found in the initial 0.12 release, the
current release is actually 0.12.1
If you have any feedback about this release or just want some help, the
best way to interact with Beaker developers and users is
the #beaker IRC channel on irc.freenode.net. Alternatively, you can post
your question to the beaker-devel mailing list [4].
[1] http://beaker-project.org/download.html
[2] http://beaker-project.org/docs/whats-new/release-0.12.html
[3] http://beaker-project.org/help.html
[4] https://fedorahosted.org/mailman/listinfo/beaker-devel
[5] http://beaker-project.org/releases/beaker-0.12.1-ChangeLog
--
Nick Coghlan
Red Hat Infrastructure Engineering & Development, Brisbane
Python Applications Team Lead
Beaker Development Lead (http://beaker-project.org/)
PulpDist Development Lead (http://pulpdist.readthedocs.org)
10 years, 11 months
Re: [Beaker-devel] [RFC V3 PATCH 0/5] beaker harness support for autotest
by Dan Callaghan
Excerpts from Don Zickus's message of 2013-04-13 05:09:48 +1000:
> Hi Folks,
>
> Throwing this over the wall for a new round of reviewing.
Thanks for all your work on these patches, Don!
I had a look through them all this afternoon (I'm sorry it's taken me so
long to get to it). I see there is a fair bit of code in there to
support running existing Beaker/RHTS-style tasks inside autotest
(emulating rhts-* commands, etc). I didn't realise that before.
Using autotest to run existing Beaker tasks is interesting and may be
worth pursuing, but what's *more* interesting to us is the ability to
run autotest tests as part of a Beaker recipe. We've had some interest
from people who have existing autotest-based tests who would like to
make use of Beaker's scheduling abilities.
Given my very limited understanding of autotest, this is what I was
imagining we would need to do:
* Find a way to map native autotest control files to Beaker recipe XML.
The control files are real Python code, which makes them very
powerful, so I don't think it makes sense to try to abstract that away
in Beaker's XML. It might make more sense to allow the control file to
be embedded literally in the recipe XML. For example we could add
support for something like this:
<recipe>
...
<script><![CDATA[
job.run_test(...)
]]></script>
</recipe>
where <script/> replaces the series of <task/> elements that you would
normally see in a Beaker recipe.
* Write some code for autotest to fetch the Beaker recipe XML and
configure itself accordingly (that's the bootstrap stuff in these
patches).
* Wrote some code for autotest to collect test output and results and
report them to Beaker somehow (I don't see anything for this in these
patches, aside from the rhts-submit-log/rhts-report-result emulation?)
It would be great if we could get this working first, and then the
support for running RHTS-style tasks could be an extra patch set on top
of that.
Don, how does that sound to you?
--
Dan Callaghan <dcallagh(a)redhat.com>
Software Engineer, Infrastructure Engineering and Development
Red Hat, Inc.
10 years, 11 months
Re: [Beaker-devel] [RFC V3 PATCH 0/5] beaker harness support for autotest
by Lucas Meneghel Rodrigues
On Fri, 2013-04-12 at 15:09 -0400, Don Zickus wrote:
> Hi Folks,
FYI,
I've transformed your patchset into a github pull request, so you see
how one looks. Of course, I'll comment on the emails themselves in case
we find problems with it, so don't worry about getting out of the loop,
OK?
https://github.com/autotest/autotest/pull/629
Cheers,
Lucas
> Throwing this over the wall for a new round of reviewing.
>
> Recently a bunch of autotest and beaker folks have been discussing how to get
> an autotest client working under beaker (namely Nick, Dan, Bill from beaker and
> Lucas, Cleber, Ademar from autotest).
>
> A long time ago Jan Stancek whipped up some code to do that based on some
> discussions we both had. I later modified a bunch of his code and came up with
> these patches.
>
> I lasted touched these patches back in August 2012. I never finished polishing
> them off or completed my testing due to other Red Hat obligations.
>
> I am presenting these patches as a starting point for discussions as to how Jan
> and myself solved various problems. They are very much 'beta' patches (there
> has been a lot of testing so they were stable at one point).
>
> Unfortunately, I don't remember how all the pieces worked so instead of
> breaking the files into digestable chunks, I am just submitting whole files as
> new (well the first patch is a 'patch').
>
> Please be kind. I am cross-posting to various mailing lists to let anyone
> interested participate.
>
> The end goal is to have autotest client work with the beaker server smoothly.
> This would allow various autotest projects to leverage the
> provisioning/inventorying mechanisms provided by beaker for various labs.
>
> Feel free to direct all questions to me and I will do my best to remember what
> Jan and myself did.
>
> I started running our internal kernel testsuite to get more live coverage of these
> changes. Things work for the most part. Reboot still has a bunch of issues,
> namely the 'autotest --continue' needs work done. :-(
>
> I add some initial support to handle legacy beaker commands to talk with the
> server. I expect to simplify it later.
>
> v2: re-coded to use beaker's new stable API
> thinned things out, added support for debug env
> more testing
> cleaned up coding style and syntax based on Lucas's feedback
>
> v3: cleanup based on Lucas's feedback
> reworked file uploading to deal with reboot state
> little fixes from more testing
>
> Don Zickus (4):
> cmdparser: add new command bootstrap
> harness: support for beaker's xml
> harness: support beaker proxy protocol
> harness: new beaker harness
>
> client/autotest_local.py | 2 +-
> client/bkr_proxy.py | 418 +++++++++++++++++++++++++++++++++++++++++
> client/bkr_xml.py | 145 +++++++++++++++
> client/cmdparser.py | 77 +++++++-
> client/harness_beaker.py | 463 ++++++++++++++++++++++++++++++++++++++++++++++
> client/shared/error.py | 4 +
> 6 files changed, 1098 insertions(+), 11 deletions(-)
> create mode 100644 client/bkr_proxy.py
> create mode 100644 client/bkr_xml.py
> create mode 100644 client/harness_beaker.py
>
> Don Zickus (5):
> cmdparser: add new command bootstrap
> harness: support for beaker's xml
> harness: support beaker proxy protocol
> harness: new beaker harness
> Harness: Add beaker hook for quick commands
>
> client/autotest_local.py | 2 +-
> client/bkr_proxy.py | 409 +++++++++++++++++++++++++++++++
> client/bkr_xml.py | 145 +++++++++++
> client/cmdparser.py | 90 ++++++-
> client/harness_beaker.py | 587 +++++++++++++++++++++++++++++++++++++++++++++
> client/rhts-reboot | 4 +
> client/rhts-report-result | 7 +
> client/rhts-submit-log | 6 +
> client/rhts_submit_log | 6 +
> client/shared/error.py | 4 +
> 10 files changed, 1249 insertions(+), 11 deletions(-)
> create mode 100644 client/bkr_proxy.py
> create mode 100644 client/bkr_xml.py
> create mode 100644 client/harness_beaker.py
> create mode 100755 client/rhts-reboot
> create mode 100755 client/rhts-report-result
> create mode 100755 client/rhts-submit-log
> create mode 100755 client/rhts_submit_log
>
10 years, 11 months
Re: [Beaker-devel] [RFC V3 PATCH 0/5] beaker harness support for autotest
by Lucas Meneghel Rodrigues
On Fri, 2013-04-12 at 15:09 -0400, Don Zickus wrote:
> Hi Folks,
>
> Throwing this over the wall for a new round of reviewing.
>
> Recently a bunch of autotest and beaker folks have been discussing how to get
> an autotest client working under beaker (namely Nick, Dan, Bill from beaker and
> Lucas, Cleber, Ademar from autotest).
>
> A long time ago Jan Stancek whipped up some code to do that based on some
> discussions we both had. I later modified a bunch of his code and came up with
> these patches.
>
> I lasted touched these patches back in August 2012. I never finished polishing
> them off or completed my testing due to other Red Hat obligations.
>
> I am presenting these patches as a starting point for discussions as to how Jan
> and myself solved various problems. They are very much 'beta' patches (there
> has been a lot of testing so they were stable at one point).
>
> Unfortunately, I don't remember how all the pieces worked so instead of
> breaking the files into digestable chunks, I am just submitting whole files as
> new (well the first patch is a 'patch').
>
> Please be kind. I am cross-posting to various mailing lists to let anyone
> interested participate.
>
> The end goal is to have autotest client work with the beaker server smoothly.
> This would allow various autotest projects to leverage the
> provisioning/inventorying mechanisms provided by beaker for various labs.
>
> Feel free to direct all questions to me and I will do my best to remember what
> Jan and myself did.
>
> I started running our internal kernel testsuite to get more live coverage of these
> changes. Things work for the most part. Reboot still has a bunch of issues,
> namely the 'autotest --continue' needs work done. :-(
>
> I add some initial support to handle legacy beaker commands to talk with the
> server. I expect to simplify it later.
>
> v2: re-coded to use beaker's new stable API
> thinned things out, added support for debug env
> more testing
> cleaned up coding style and syntax based on Lucas's feedback
>
> v3: cleanup based on Lucas's feedback
> reworked file uploading to deal with reboot state
> little fixes from more testing
Thanks Don, I'll be checking this new version Real Soon (TM).
Cheers,
Lucas
10 years, 11 months
Re: [Beaker-devel] [RFC PATCH 3/4] harness: support beaker proxy protocol
by Lucas Meneghel Rodrigues
On 04/02/2013 06:18 PM, Don Zickus wrote:
> This patch contains the pieces to communicate with Beaker's new stable API.
>
> It is a thin layer that takes commands from the harness and translates them into
> the appropriate http GET, POST or PUT command.
>
> One thing I did to make debugging easier is to allow the ability to write to
> files locally (offline mode). The harness doesn't care where the data is sent,
> so writing locally allow me to quickly test and address problems without complicating
> it with broken http requests.
>
> I also implemented chunk uploading to minimize memory usage.
>
> I spent most of my time testing the log uploading and task result reporting
> aspect of the API. Everything seems to work great. I expect the 'status'
> and 'watchdog' parts to just work.
>
> Mostly of interest to the beaker folks. Autotest folks can review for style,
> syntax and entertainment.
>
> Signed-off-by: Don Zickus <dzickus(a)redhat.com>
> ---
> client/bkr_proxy.py | 418 +++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 files changed, 418 insertions(+), 0 deletions(-)
> create mode 100644 client/bkr_proxy.py
>
> diff --git a/client/bkr_proxy.py b/client/bkr_proxy.py
> new file mode 100644
> index 0000000..03b2fdc
> --- /dev/null
> +++ b/client/bkr_proxy.py
> @@ -0,0 +1,418 @@
> +# bkr_proxy.py
> +#
> +# Copyright (C) 2011 Jan Stancek <jstancek(a)redhat.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +
> +# Started by Jan Stancek <jstancek(a)redhat.com> 2011
> +"""
> +bkr_proxy - class used to talk to beaker
> +"""
> +__author__ = """Don Zickus 2013"""
> +
> +
> +import time
> +import os
> +import sys
> +import base64
> +import xmlrpclib
> +import random
> +import logging
> +import shutil
> +import httplib
> +import time
> +import re
> +import urllib, urllib2
> +from types import *
> +from autotest.client.shared import utils
There are a bunch of unused modules here. More comments below:
> +log = logging
> +
> +
> +AUTOTEST_CACHE_DIR = '/var/cache/autotest'
> +
> +class BkrProxyException(Exception):
> + def __init__(self, text):
> + Exception.__init__(self, text)
> +
> +'''Hard coded internal paths'''
> +def make_path_cmdlog(r):
> + """
> + Converts a recipe id into an internal path for logging purposes
> +
> + @param r: recipe id
> +
> + @return a path to the internal command log
> + """
> +
> + path = AUTOTEST_CACHE_DIR + '/recipes/' + r
> + if not os.path.exists(path):
> + os.makedirs(path)
> + if not os.path.isdir(path):
> + raise BkrProxyException("Path(%s) exists and is not a directory" % path)
> + return path + '/cmd_log'
> +
> +def make_path_bkrcache(r):
> + """
> + Converts a recipe id into an internal path for cache'ing recipe
> +
> + @param r: recipe id
> +
> + @return a path to the internal recipe cache file
> + """
> +
> + return AUTOTEST_CACHE_DIR + '/recipes/' + r + '/beaker_recipe.cache'
> +
> +'''End Hard coded internal paths'''
> +
> +"""
> +Hard coded paths as described in the Beaker Server API
> +http://beaker-project.org/dev/proposals/harness-api.html
> +"""
> +
> +def make_path_recipe(r):
> + """
> + Converts a recipe id into a beaker path
> +
> + @param r: recipe id
> +
> + @return a beaker path to the recipe id
> + """
> +
> + return '/recipes/' + r
> +
> +def make_path_watchdog(r):
> + """
> + Converts a recipe id into a beaker path for the watchdog
> +
> + @param r: recipe id
> +
> + @return a beaker path of the recipe's watchdog file
> + """
> +
> + return '/recipes/' + r + '/watchdog'
> +
> +def make_path_status(r, t=None):
> + """
> + Converts id into a beaker path to status file
> +
> + Given a recipe id and/or a task id, translate them into
> + the proper beaker path to the status file. Recipe only, returns
> + the path to the recipe's status, whereas including a task returns
> + the path to the task's status.
> +
> + @param r: recipe id
> + @param t: task id
> +
> + @return a beaker path of the recipe's/task's status file
> + """
> +
> + rpath = '/recipes/' + r
> + tpath = t and '/tasks/' + t or ''
> +
> + return rpath + tpath + '/status'
> +
> +def make_path_result(r, t):
> + """
> + Converts task id into a beaker path to result file
> +
> + Given a recipe id and a task id, translate them into
> + the proper beaker path to the result file.
> +
> + @param r: recipe id
> + @param t: task id
> +
> + @return a beaker path of the task's result file
> + """
> +
> + rpath = '/recipes/' + r
> + tpath = '/tasks/' + t
> +
> + return rpath + tpath + '/results/'
> +
> +def make_path_log(r, t=None, i=None):
> + """
> + Converts id into a beaker path to log file
> +
> + Given a recipe id, a task id, and/or result id, translate
> + them into the proper beaker path to the log file. Depending
> + on which log file is needed, provide the appropriate params.
> + Note the dependency, a result id needs a task id and recipe id,
> + while a task id needs a recipe id.
> +
> + @param r: recipe id
> + @param t: task id
> + @param i: result id
> +
> + @return a beaker path of the task's result file
> + """
> +
> + rpath = '/recipes/' + r
> + tpath = t and '/tasks/' + t or ''
> + ipath = i and '/results/' + i or ''
> +
> + return rpath + tpath + ipath + '/logs'
> +
> +'''End Hard coded paths'''
> +
> +def copy_remote(data, dest, use_put=None):
> + """
> + Copy data to a remote server using http calls POST or PUT
> +
> + Using http POST and PUT methods, copy data over http. To use
> + PUT method, provide a dictionary of values to be populated in
> + the Content-Range and Content-Length headers. Otherwise default
> + is to use POST method.
> +
> + Traps on HTTPError 500 and 400
> +
> + @param data: encoded data string to copy remotely
> + @param dest: remote server URL
> + @param use_put: dictionary of items if using PUT method
> +
> + @return html header info for post processing
> + """
> +
> + ret = None
> + req = urllib2.Request(dest, data=data)
> + if use_put:
> + req.add_header('Content-Type', 'application/octet-stream')
> + end = use_put['start'] + use_put['size'] - 1
> + req.add_header('Content-Range', 'bytes %s-%s/%s' % (use_put['start'],
> + end, use_put['total']))
> + req.add_header('Content-Length', '%s' % use_put['size'])
> + req.get_method = lambda: 'PUT'
> +
> + try:
> + res = utils.urlopen(req)
> + ret = res.info()
> + res.close()
> + except urllib2.HTTPError as e:
^ This construct was introduced in py 2.5 and will break things when
running on py 2.4. As we strive to keep compatibility with it, you've
got to replace this with a comma:
except urllib2.HTTPError, e:
I did fix all of the little problems I found and send them to your
email, so you can apply them and re-send the patchset.
10 years, 11 months
Re: [Beaker-devel] [Autotest] [RFC PATCH 4/4] harness: new beaker harness
by Lucas Meneghel Rodrigues
On 04/02/2013 06:18 PM, Don Zickus wrote:
> This is the bulk of the harness file for enabling beaker.
>
> The idea is to bootstrap it by downloading the remote xml file, parse it,
> convert it into a control file and pass that to autotest.
>
> The rest of the harness has various hacks to deal with passing info
> from autotest to beaker.
>
> Two of the biggest pieces are converting the xml recipe into a control
> file. This will still need tweaking.
>
> The other piece is trying to figure out when recipe/tasks/sub-tasks
> start and stop. Again probably needs tweaking.
>
> Overall, this patch has been tested with some stub recipes and responds
> well. Work still needs to be done on figuring out which files to
> upload and real workflow testing.
>
> This patch will need to be reviewed by autotest and beaker folks.
>
> Signed-off-by: Don Zickus <dzickus(a)redhat.com>
> ---
> client/harness_beaker.py | 463 ++++++++++++++++++++++++++++++++++++++++++++++
> 1 files changed, 463 insertions(+), 0 deletions(-)
> create mode 100644 client/harness_beaker.py
>
> diff --git a/client/harness_beaker.py b/client/harness_beaker.py
> new file mode 100644
> index 0000000..e901fab
> --- /dev/null
> +++ b/client/harness_beaker.py
> @@ -0,0 +1,463 @@
> +# harness_beaker.py
> +#
> +# Copyright (C) 2011 Jan Stancek <jstancek(a)redhat.com>
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
> +#
> +# started by Jan Stancek <jstancek(a)redhat.com> 2011
> +"""
> +The harness interface
> +The interface between the client and beaker lab controller.
> +"""
> +__author__ = """Don Zickus 2013"""
> +
> +import os
> +import optparse
> +import logging as log
> +import harness
> +import time
> +import re
> +import sys
> +from time import gmtime, strftime
> +from autotest.client.shared import utils,error
> +
> +from autotest.client.bkr_xml import BeakerXMLParser
> +from autotest.client.bkr_proxy import BkrProxy
> +
> +'''Use 5 minutes for console heartbeat'''
> +BEAKER_CONSOLE_HEARTBEAT = 60 * 5
> +
> +class harness_beaker(harness.harness):
> + def __init__(self, job, harness_args):
> + log.debug('harness_beaker __init__')
> + super(harness_beaker, self).__init__(job)
> +
> + #temporary hack until BEAKER_RECIPE_ID and BEAKER_LAB_CONTROLLER_URL is setup in beaker
> + os.environ['BEAKER_RECIPE_ID'] = open('/root/RECIPE.TXT', 'r').read().strip()
> + os.environ['BEAKER_LAB_CONTROLLER_URL'] = re.sub("/bkr/", ":8000", os.environ['BEAKER'])
> +
> + #control whether bootstrap environment remotely connects or stays offline
> + #cheap hack to support flexible debug environment
> + #the bootstrap job object is just a stub and won't have the '_state' attribute
> + if hasattr(job, '_state'):
> + is_bootstrap = False
> + else:
> + is_bootstrap = True
> +
> + self.state_file = os.path.join(os.path.dirname(__file__), 'harness_beaker.state')
> + self.recipe_id = os.environ.get('BEAKER_RECIPE_ID')
> + self.labc_url = os.environ.get('BEAKER_LAB_CONTROLLER_URL')
> + self.hostname = os.environ.get('HOSTNAME')
> + #self.common_results_uploaded_after = False
> + self.watchdog_pid = None
> + self.current_task_id = None
> + self.skip_upload = False #hack to work around reservesys
> + self.offline = False
> +
> + if harness_args:
> + log.info('harness_args: %s' % harness_args)
> + self.args = self.parse_args(harness_args, is_bootstrap)
> +
> +
> + log.debug('harness_beaker: state_file: <%s>', self.state_file)
> +
> + log.debug('harness_beaker: hostname: <%s>', self.hostname)
> + log.debug('harness_beaker: labc_url: <%s>', self.labc_url)
> +
> + if not self.hostname:
> + raise error.HarnessError('Need valid hostname')
> +
> + #hack for flexible debug environment
> + labc = not self.offline and self.labc_url or None
> +
> + self.bkr_proxy = BkrProxy(self.recipe_id, labc)
> +
> + #self.setupInitSymlink()
> + #self.setupAutotestConfFile()
> +
> + def parse_args(self, args, is_bootstrap):
> + if not args:
> + return
> +
> + for a in args.split(','):
> + if a == 'offline':
> + #use cached recipe and stay offline whole time
> + self.offline = True
> +
> + elif a[:5] == 'cache':
> + if len(a) > 5 and a[5] == '=':
> + #cache a different recipe instead
> + self.recipe_id = a[6:]
> +
> + #remotely retrieve recipe, but stay offline during run
> + if not is_bootstrap:
> + self.offline = True
> +
> + else:
> + raise error.HarnessError("Unknown beaker harness arg: %s" % a)
> +
> + def bootstrap(self, fetchdir):
> + '''How to kickstart autotest when you have no control file?
> + You download the beaker XML, convert it to a control file
> + and pass it back to autotest. Much like bootstrapping.. :-)
> + '''
> +
> + recipe = self.init_recipe_from_beaker()
> +
> + #sanity check
> + if self.recipe_id != recipe.id:
> + raise error.HarnessError('Recipe mismatch: machine %s.. != XML %s..' %
> + (self.recipe_id, recipe.id))
> +
> + #create unique name
> + control_file_name = recipe.job_id + '_' + recipe.id + '.control'
> + control_file_path = fetchdir + '/' + control_file_name
> +
> + log.debug('setting up control file - %s' % control_file_path)
> + control_file = open(control_file_path, 'w')
> + try:
> + #convert recipe xml into control file
> + for task in recipe.tasks:
> + self.convert_task_to_control(fetchdir, control_file, task)
> + control_file.close()
> + except Exception, ex:
> + os.remove(control_file_path)
> + raise error.HarnessError('beaker_harness: convert failed with -> %s' % ex)
> +
> + #autotest should find this under FETCHDIRTEST because it is unique
> + return control_file_path
> +
> + def init_recipe_from_beaker(self):
> + log.debug('Contacting beaker to get task details')
> + bxp = BeakerXMLParser()
> + recipe_xml = self.get_recipe_from_LC()
> + recipes_dict = bxp.parse_xml(recipe_xml)
> +
> + return self.find_recipe(recipes_dict)
> +
> + def init_task_params(self, task):
> + log.debug('PrepareTaskParams')
> + if task == None:
> + raise error.HarnessError('No valid task')
> +
> + for (name, value) in task.params.items():
> + log.debug('adding to os.environ: <%s=%s>', name, value)
> + os.environ[name] = value
> +
> + def get_recipe_from_LC(self):
> + log.debug('trying to get recipe from LC:')
> + try:
> + recipe = self.bkr_proxy.get_recipe()
> + except Exception, exc:
> + raise error.HarnessError('Failed to retrieve xml: %s' % exc)
> + return recipe
> +
> + def find_recipe(self, recipes_dict):
> + if self.hostname in recipes_dict:
> + return recipes_dict[self.hostname]
> + for h in recipes_dict:
> + if self.recipe_id == recipes_dict[h].id:
> + return recipes_dict[h]
> + raise error.HarnessError('No valid recipe for host %s' % self.hostname)
> +
> + def get_test_name(self, task):
> + name = re.sub('-','_', task.rpmName)
> + return re.sub('\.','_', name)
> +
> + def convert_task_to_control(self, fetchdir, control, task):
> + """Tasks are really just:
> + # yum install $TEST
> + # cd /mnt/tests/$TEST
> + # make run
> +
> + Convert that into a test module with a control file
> + """
> + timeout = ''
> + if task.timeout:
> + timeout = ", timeout=%s" % task.timeout
> +
> + #python doesn't like '-' in its class names
> + rpm_name = self.get_test_name(task)
> + rpm_dir = fetchdir + '/' + rpm_name
> + rpm_file = rpm_dir + '/' + rpm_name + '.py'
> +
> + if not self.offline and task.status == 'Completed':
> + log.debug("SKIP Completed test %s" % rpm_name)
> + return
> +
> + if task.status == 'Running' and not re.search('reservesys', task.rpmName):
> + log.warning("Found Running test %s that isn't reservesys" % task.rpmName)
> +
> + #append test name to control file
> + log.debug('adding test %s to control file' % rpm_name)
> +
> + #Trick to avoid downloading XML all the time
> + #statically update each TASK_ID
> + control.write("os.environ['BEAKER_TASK_ID']='%s'\n" % task.id)
> + control.write("job.run_test('%s'%s)\n" % (rpm_name, timeout))
> +
> + #TODO check for git commands in task.params
> +
> + #create the test itself
> + log.debug('setting up test %s' % (rpm_file))
> + if not os.path.exists(rpm_dir):
> + os.mkdir(rpm_dir)
> + test = open(rpm_file, 'w')
> + test.write("import os\n")
> + test.write("from autotest.client import test, utils\n\n")
> + test.write("class %s(test.test):\n" % rpm_name)
> + test.write(" version=1\n\n")
> + test.write(" def setup(self):\n")
> + test.write(" utils.system('yum install -y %s')\n" % task.rpmName)
> + for param in task.params:
> + print "param is %s" % param
> + test.write(" os.environ['%s']='%s'\n" % (param, task.params[param]))
> + test.write(" def run_once(self):\n")
> + test.write(" os.chdir('%s')\n" % task.rpmPath)
> + #test.write(" raw_output = utils.system_output('make run', retain_output=True)\n")
> + test.write(" raw_output = utils.system_output('make help', retain_output=True)\n")
> + test.write(" self.results = raw_output\n")
> + if rpm_name == '/distribution/reservesys':
> + test.write(" self.job.harness.skip_upload=True\n")
> + test.close()
> +
> + def run_start(self):
> + """A run within this job is starting"""
> + log.debug('run_start')
> + try:
> + self.start_watchdog(BEAKER_CONSOLE_HEARTBEAT)
> + except Exception, exc:
> + log.critical('ERROR: Failed to start watchdog')
> +
> + def run_pause(self):
> + """A run within this job is completing (expect continue)"""
> + log.debug('run_pause')
> +
> + def run_reboot(self):
> + """A run within this job is performing a reboot
> + (expect continue following reboot)
> + """
> + log.debug('run_reboot')
> +
> + def run_abort(self):
> + """A run within this job is aborting. It all went wrong"""
> + log.debug('run_abort')
> + self.bkr_proxy.recipe_abort()
> + self.tear_down()
> +
> + def run_complete(self):
> + """A run within this job is completing (all done)"""
> + log.debug('run_complete')
> + self.tear_down()
> +
> + def run_test_complete(self):
> + """A test run by this job is complete. Note that if multiple
> + tests are run in parallel, this will only be called when all
> + of the parallel runs complete."""
> + log.debug('run_test_complete')
> +
> + def test_status(self, status, tag):
> + """A test within this job is completing"""
> + log.debug('test_status ' + status + ' / ' + tag)
> +
> + def test_status_detail(self, code, subdir, operation, status, tag,
> + optional_fields):
> + """A test within this job is completing (detail)"""
> +
> + log.debug('test_status_detail %s / %s / %s / %s / %s / %s',
> + code, subdir, operation, status, tag, str(optional_fields))
> +
> + """The mapping between a beaker recipe/task to an autotest test is not
> + always clear because of various reasons. One is the first/last
> + START/END maps to a recipe. The next set of START/END maps to a task.
> + But the set after might map to a sub-test not in beaker thus needs to
> + be recorded as a sub-task. Or it could map to the next task in the XML.
> +
> + So we control this by using START/END as boundary markers and an 'id' stack
> + to let us know wheter we are starting/finishing a recipe, task or sub-task.
> +
> + Based on this info, we know which beaker 'status' API to use and where to
> + upload 'results'
> +
> + This is entirely hinged on the environment variable BEAKER_TASK_ID being
> + updated correctly in the control file. If it isn't being updated correctly,
> + then everything may look like a sub-task and beaker gets out of sync.
> +
> + If that is the case, running the harness 'sync' command should re-write the
> + control file based on where beaker thinks we should be and all should be
> + good again.
> + """
> + if code.startswith('START'):
> + '''Task ID not setup in first START'''
> + if self.current_task_id:
> + if 'BEAKER_TASK_ID' not in os.environ:
> + raise error.HarnessError("No BEAKER_TASK_ID set")
> + task_id = os.environ['BEAKER_TASK_ID']
> +
> + if self.current_task_id is None:
> + '''First START'''
> + log.debug('START: Pushing recipe id (%s) onto stack' % self.recipe_id)
> + self.current_task_id = [self.recipe_id]
> + elif self.current_task_id and self.current_task_id != task_id:
> + '''New beaker task START'''
> + log.debug('START: Pushing task id (%s) onto stack' % task_id)
> + self.current_task_id.append(task_id)
> + self.bkr_proxy.task_start(task_id)
> + else:
> + '''Sub task not known to beaker'''
> + log.debug('START: Pushing sub-task (0) onto stack')
> + self.current_task_id.append(0)
> +
> + return
> + elif code.startswith('END'):
> + task_id = self.current_task_id.pop()
> +
> + if task_id == self.recipe_id:
> + '''Last END'''
> + log.debug('END: Popping recipe id (%s) off stack' % task_id)
> + self.bkr_proxy.recipe_stop()
> + elif task_id:
> + '''Valid Beaker task id, cleanup'''
> + log.debug('END: Popping task id (%s) off stack' % task_id)
> + self.bkr_proxy.task_stop(task_id)
> + #self.upload_task_files()
> + #FIXME upload task log files
> + else:
> + '''Sub task not known to beaker, skip'''
> + log.debug('END: Popping sub-task (0) off stack')
> + #FIXME upload sub-task results
> + #FIXME report sub-task Status
> +
> + return
> + else:
> + '''Normal sub-task completions'''
> + bkr_status = get_beaker_code(code)
> + task_id = self.current_task_id.pop()
> + self.current_task_id.append(task_id)
> + try:
> + resultid = self.bkr_proxy.task_result(task_id, bkr_status,
> + subdir, 1, '')
> + self.upload_test_results(task_id, resultid, subdir)
> + except Exception, exc:
> + log.critical('ERROR: Failed to process test results')
> +
> + def tear_down(self):
> + '''called from complete and abort. clean up and shutdown'''
> + #self.recipe_upload_files(self.recipe_xml.id)
> + self.kill_watchdog()
> + pass
> +
> + def start_watchdog(self, heartbeat):
> + log.debug('harness: Starting watchdog process, heartbeat: %d' % heartbeat)
> + try:
> + pid = os.fork()
> + if pid == 0:
> + self.watchdog_loop(heartbeat)
> + else:
> + self.watchdog_pid = pid
> + log.debug('harness: Watchdog process started, pid: %d', self.watchdog_pid)
> + except OSError, e:
> + log.error('harness: fork in start_watchdog failed: %d (%s)\n' % (e.errno, e.strerror))
> +
> + def kill_watchdog(self):
> + log.debug('harness: Killing watchdog, pid: %d', self.watchdog_pid)
> + utils.nuke_pid(self.watchdog_pid)
> + self.watchdog_pid = None
> +
> + def watchdog_loop(self, heartbeat):
> + while True:
> + time.sleep(heartbeat)
> + log.info('[-- MARK -- %s]' % time.asctime( time.localtime(time.time())))
> + sys.exit()
> +
> + def get_processed_tests(self):
> + subdirs = []
> + if os.path.isfile(self.state_file):
> + f = open(self.state_file, 'r')
> + lines = f.readlines()
> + f.close()
> + for line in lines:
> + subdirs.append(line.strip())
> + return subdirs
> +
> + def write_processed_tests(self, subdirs):
> + f = open(self.state_file, 'w')
> + for subdir in subdirs:
> + f.write(subdir + '\n')
> + f.close()
> +
> + # filter our some of the dirs, because this one is called at start/stop of test
> + def upload_common_results(self):
> + try:
> + log.debug('Uploading common test results')
> + filter_dirs = self.get_processed_tests()
> + filter_dirs.extend(['analysis', 'sysinfo'])
> +
> + #log.info('sleeping for 3 seconds')
> + #time.sleep(3)
> +
> + self.upload_task_files(filter_dirs)
> + except Exception, exc:
> + log.critical('ERROR: Failed to upload task results')
> +
> + def upload_task_files(self, except_dirs):
> + log.debug('Uploading results except dirs: ' + str(except_dirs))
> + task = self.recipe_xml.task[0]
> + path = self.job.resultdir + '/' + self.getTestName(task)
> +
> + for root, dirnames, files in sorted(os.walk(path)):
> + for name in files:
> + #strip full path
> + basedir = re.sub(path + "/", "", root)
> + result_file = root + '/' + name
> + self.bkr_proxy.task_upload_file(result_file,
> + task.id, name, basedir)
> +
> + '''do not upload previously uploaded results files'''
> + for edir in task.excluded_dir:
> + if edir in dirnames:
> + dirnames.remove(edir)
> +
> + '''add test name in future excluded directories for recipe'''
> + self.recipe_xml.excluded_dir.append(self.getTestName(task))
Here in the method above you forgot to update the calls to the
task_upload_file API, which has changed when you rewrote bkr_proxy. I
took a shot at trying to fix the function, check it out:
> def upload_task_files(self, except_dirs):
> - log.debug('Uploading results except dirs: ' + str(except_dirs))
> + log.debug('Uploading results except dirs: %s', str(except_dirs))
> task = self.recipe_xml.task[0]
> - path = self.job.resultdir + '/' + self.getTestName(task)
> + path = os.path.join(self.job.resultdir, self.getTestName(task))
>
> for root, dirnames, files in sorted(os.walk(path)):
> for name in files:
> - #strip full path
> + # Strip the base dir (path) for the remote path
> basedir = re.sub(path + "/", "", root)
> - result_file = root + '/' + name
> - self.bkr_proxy.task_upload_file(result_file,
> - task.id, name, basedir)
> + remotepath = os.path.join(basedir, name)
> + # The localfile has the full path
> + localfile = os.path.join(root, name)
> + # Upload the file
> + self.bkr_proxy.task_upload_file(task.id, localfile,
> + remotepath)
I *think* this is right, it was just something that came up with the
basic checks I usually put the patchsets through. Other than this small
mistake, I found some trailing whitespace errors and other little
indentation problems. Would you please fix this real quick, use
utils/reindent.py to reindent both client/harness_beaker.py and
client/bkr_proxy.py and send an updated patchset? Thanks!
10 years, 11 months