Test Harness status - 2009-07-15
by Marian Csontos
Hi Bill, hi Kevin,
I ran into problems with my model on Friday and realised it was too
simplistic.
I had to rethink few things and this is the result.
--------------------------------------------------------------------------------
== TEST HARNESS ==
=== GENERAL OVERVIEW ===
+-------------------+-------------
| BACKEND (LC) | REMOTE/LOCAL
+vv^^^vv^v^^vvv^v^^^+-------------
| HARNESS |
+^^vvvv^^^^vvvv^v^^v+ LOCAL
| TEST |
+-------------------+-------------
BACKEND is all the infrastructure for scheduling, collecting logs and
results etc.
In Beaker this is represented by Lab Controller, which delegates part of the
tasks to Scheduler.
TEST is a test script. In general it could be any script which conforms
to some
rules. There is not a major player in this area - there are many unit test
frameworks, Perl Test::Harness::TAP format, STAF,...
HARNESS is a tool for connecting these two parts. Its main tasks are to:
* ask BACKEND to provide the TEST to run,
* install and execute the TEST,
* capture and analyze the TEST's output and
* report to BACKEND.
In this model there are two interfaces, where we need an adaptation layers:
- interface to backend(LC)
- and test script itself (using test scripts non conforming to interface
specifiaction)
=== MODEL ===
+v^^vvvv^^^^vv^vvv^vvv^^^^^vv^v^^^^vv^+---------
| server-adaptor |
+--------------------+----------------+
| | controller-lib | HARNESS
| +----------------+
| test-controller |
+------------+-------------+----------+---------
|test-adaptor| harness-lib | |
+vv^^^v^vvv^v+-------------+ | TASK
| TEST |
+-------------------------------------+---------
The whole stack is split into two parts - HARNESS and TASK
HARNESS is task independent part - this will be the same regardless of
type of
test, and where possible platform/architecture independent.
TASK is any script/executable compatible with HARNESS interface.
==== HARNESS ====
HARNESS will be installed and set-up during provisioning phase (or as
packages
by developer in case of manual execution)
* test-controller - the heart of HARNESS - download, install, run, log and
report. test-controller has to be able running multiple tasks at the same
time and handle concurrent input from them.
* controller-lib - system/platform dependent part
* system dependent set-up
* system dependent routines - e.g. use yum/up2date or wget?
* server-adaptor - the only backend(LC) dependent module.
(beaker|local|...)
* there will be more adaptors allowed at the same time - e.g. one
to server, another for displaying results locally, and one to store
them
directly in DB. Local adapters can be attached/detached.
==== TASK ====
TASK will be downloaded and installed by HARNESS (or developer in case of
manual execution)
* TEST was described in general overview
* test-adaptor is adaptation layer conformant to HARNESS API. It is the
only
TEST dependent component (apart from the test itself.) It will
* install, setup, execute the script
* capture and analyse output of TEST and report back to HARNESS
* harness-lib provides language bindings to HARNESS API for
harness-aware test
scripts. It is the only test implementation language dependent component.
Library will provide functions for logging, file-uploading, streaming,
result-sumbitting, task-spawning, synchronizing, rebooting...
=== FOOTNOTES ===
==== System reboot handling ====
TASK should not be allowed to reboot the system, as this would affect all
concurrently runing tasks. It has to signal harness, which will suspend
reboot
untill it is safe. All the running TASKS has to be cooperative, or we would
have to wait untill the task finishes.
==== Meta-tasks ====
Task comprising of many subtasks and rules for executing them:
* Sequential|Parallel execution
* Continue when Any|At-most(N)|None tests fails
* Repeat(N) - run single task N times
Though this could be handled on Server-side, but it would not allow running
such tests locally.
==== Concurrency ====
For tasks wishing to spawn other tasks, it would be useful, if these were
registerred to harness.
=== IMPLEMENTATION ===
1. Interface between HARNESS and TASK
* Marian suggests using single named pipe with structured data
(xml-rpc? Tag-Length-Value encoded data?)
* most of the traffic from TASK to HARNESS will be simply forwarded by
test-controller to server-adaptor.
2. Implement server-adaptor for displaying to console/redirecting to a
file.
3. Implement basic functionality of test-controller && controller lib
* install, set-up, execute, capture and report,
* no concurrency, no spawning new tasks
4. Implement a test-adaptor for Perl's Test::Harness::TAP format
* It is simple enough for evaluation
5. Run some tests
6. Go back to 1, iterate...
--------------------------------------------------------------------------------
Heh, I spent few hours sorting ideas and writing this. Should cancel
SICK DAY
and report it as 1/2 day... ;-)
Looks like it could be worthy publishing on wiki. Any suggestions for
improvements, before I do so?
Cheers,
-- Marian
14 years, 8 months
XMLRPC methods for reporting results
by Bill Peck
Hello,
I've finally implemented some of the XMLRPC methods for reporting
results into the scheduler.
recipes.Abort(recipe_id, msg)
recipes.Cancel(recipe_id, msg)
recipes.tasks.Start(task_id)
recipes.tasks.Stop(task_id)
recipes.tasks.Pass(task_id, path, score, summary)
recipes.tasks.Warn(task_id, path, score, summary)
recipes.tasks.Fail(task_id, path, score, summary)
recipes.tasks.Panic(task_id, path, score, summary)
recipes.tasks.Abort(task_id, msg)
recipes.tasks.Cancel(task_id, msg)
Any status or result reported will propagate the status and result with
the highest severity to the top.
Here are the current Status and the matching severity.
New 10
Processed 20
Queued 30
Scheduled 40
Running 80
Completed 50
Cancelled 60
Aborted 70
And Results are as follows:
New 10
Pass 20
Warn 30
Fail 40
Panic 50
I was thinking it would be nice to have a recipes.tasks.StdOut(task_id,
offset, length, data) method that would take the StdOut from the running
test as its running. We would capture StdErr as well and possibly mux
them together by default or allow for separate viewing?
14 years, 8 months
Handling Multi-host recipes
by Bill Peck
Just sending this to the list to get feedback. If you remember from
previous emails a recipe = a single system. a RecipeSet contains
mutliple recipes that need to be scheduled at the same time. Also, we
only want multi-host jobs to be scheduled on the same lab controller.
A couple problems present themselves right away.
1) If N-1 recipes are able to be scheduled on a single lab controller
then we don't want any of those possible systems since we couldn't
possibly fulfil the recipeset there.
2) If RecipeA can use HOSTA or HOSTB but RecipeB can only use HOSTA,
then we need to remove HOSTA from RecipeA's possible choices.
3) If the number of possible Systems in each Recipe is larger than the
number of Recipes in the RecipeSet Then there are enough choices to
prevent a dead lock. In other words, we can leave the choices alone.
All of the above has to be done for every lab controller the recipeSet
matches for. :-)
The following code seems to work. I have tested quite a few
possibilities and everything works out. It does seem to remove more
choices than needed in some situations but I can't see an easier way.
The good news is this is still pretty damn fast. And we only need to
do it once. (ok, if the inventory server data changes we need to do it
again), but not every 20 seconds! ;-)
I'm also attaching a small test script that I used to develop the ideas
below..
def processed_recipesets(*args):
recipesets = RecipeSet.query()\
.join(['recipes','status'])\
.filter(Recipe.status==TestStatus.by_name(u'Processed'))
for recipeset in recipesets:
bad_l_controllers = set()
# We only need to do this processing on multi-host recipes
if len(recipeset.recipes) == 1:
print "recipe ID %s moved from Processed to Queued" %
recipeset.recipes[0].id
recipeset.recipes[0].status = TestStatus.by_name(u'Queued')
continue
# Find all the lab controllers that this recipeset may run.
rsl_controllers = set(LabController.query()\
.join(['systems',
'queued_recipes',
'recipeset'])\
.filter(RecipeSet.id==recipeset.id).all())
# Any lab controllers that are not associated to all recipes in the
# recipe set must have those systems on that lab controller removed
# from any recipes. For multi-host all recipes must be schedulable
# on one lab controller
for recipe in recipeset.recipes:
rl_controllers = set(LabController.query()\
.join(['systems',
'queued_recipes'])\
.filter(Recipe.id==recipe.id).all())
bad_l_controllers =
bad_l_controllers.union(rl_controllers.difference(rsl_controllers))
for l_controller in rsl_controllers:
enough_systems = False
for recipe in recipeset.recipes:
systems = recipe.dyn_systems.filter(
System.lab_controller==l_controller
).all()
if len(systems) < len(recipeset.recipes):
break
else:
# There are enough choices We don't need to worry about dead
# locks
enough_systems = True
if not enough_systems:
# Eliminate bad choices.
for recipe in recipeset.recipes:
for tmprecipe in recipeset.recipes:
systemsa = set(recipe.dyn_systems.filter(
System.lab_controller==l_controller
).all())
systemsb = set(tmprecipe.dyn_systems.filter(
System.lab_controller==l_controller
).all())
if systemsa.difference(systemsb):
for rem_system in
systemsa.intersection(systemsb):
print "Removing %s from recipe id %s" %
(rem_system, recipe.id)
recipe.systems.remove(rem_system)
for recipe in recipeset.recipes:
count = 0
systems = recipe.dyn_systems.filter(
System.lab_controller==l_controller
).all()
for tmprecipe in recipeset.recipes:
tmpsystems = tmprecipe.dyn_systems.filter(
System.lab_controller==l_controller
).all()
if recipe != tmprecipe and \
systems == tmpsystems:
count += 1
if len(systems) <= count:
# Remove all systems from this lc on this rs.
bad_l_controllers =
bad_l_controllers.union([l_controller])
# Remove systems that are on bad lab controllers
# This means one of the recipes can be fullfilled on a lab
controller
# but not the rest of the recipes in the recipeSet.
# This could very well remove ALL systems from all recipes in this
# recipeSet. If that happens then the recipeSet cannot be scheduled
# and will be aborted by the abort process.
for recipe in recipeset.recipes:
for l_controller in bad_l_controllers:
systems = (recipe.dyn_systems.filter(
System.lab_controller==l_controller
).all()
)
for system in systems:
print "Removing %s from recipe id %s" % (system,
recipe.id)
recipe.systems.remove(system)
if recipe.systems:
# Set status to Queued
print "recipe ID %s moved from Processed to Queued" %
recipe.id
recipe.status = TestStatus.by_name(u'Queued')
else:
# Set status to Aborted
print "recipe ID %s moved from Processed to Aborted" %
recipe.id
recipe.recipeset.abort('Recipe ID %s does not match any
systems' % recipe.id)
session.flush()
#!/usr/bin/python
recipe1 = set(['a','b'])
recipe2 = set(['a','b','c','f'])
recipe3 = set(['a','b','c'])
recipe4 = set(['a','b'])
recipes = []
recipes.append(recipe1)
recipes.append(recipe2)
recipes.append(recipe3)
#recipes.append(recipe4)
for recipe in recipes:
print recipe
print "eliminate bad choices"
for i, recipe in enumerate(recipes):
for tmprecipe in recipes:
if len(tmprecipe) <= len(recipes):
if recipes[i].difference(tmprecipe):
print "Difference", recipes[i].difference(tmprecipe)
print "Intersection", recipes[i].intersection(tmprecipe)
for j in recipes[i].intersection(tmprecipe):
recipes[i].remove(j)
#recipes[i] = recipes[i].difference(tmprecipe)
for recipe in recipes:
print recipe
for i, recipe in enumerate(recipes):
count = 0
for j, tmprecipe in enumerate(recipes):
if i != j and recipe == tmprecipe:
count += 1
if len(recipe) <= count:
print "get rid of recipe", recipe
14 years, 8 months