Hi Bill, hi Kevin,
I ran into problems with my model on Friday and realised it was too
simplistic.
I had to rethink few things and this is the result.
--------------------------------------------------------------------------------
== TEST HARNESS ==
=== GENERAL OVERVIEW ===
+-------------------+-------------
| BACKEND (LC) | REMOTE/LOCAL
+vv^^^vv^v^^vvv^v^^^+-------------
| HARNESS |
+^^vvvv^^^^vvvv^v^^v+ LOCAL
| TEST |
+-------------------+-------------
BACKEND is all the infrastructure for scheduling, collecting logs and
results etc.
In Beaker this is represented by Lab Controller, which delegates part of the
tasks to Scheduler.
TEST is a test script. In general it could be any script which conforms
to some
rules. There is not a major player in this area - there are many unit test
frameworks, Perl Test::Harness::TAP format, STAF,...
HARNESS is a tool for connecting these two parts. Its main tasks are to:
* ask BACKEND to provide the TEST to run,
* install and execute the TEST,
* capture and analyze the TEST's output and
* report to BACKEND.
In this model there are two interfaces, where we need an adaptation layers:
- interface to backend(LC)
- and test script itself (using test scripts non conforming to interface
specifiaction)
=== MODEL ===
+v^^vvvv^^^^vv^vvv^vvv^^^^^vv^v^^^^vv^+---------
| server-adaptor |
+--------------------+----------------+
| | controller-lib | HARNESS
| +----------------+
| test-controller |
+------------+-------------+----------+---------
|test-adaptor| harness-lib | |
+vv^^^v^vvv^v+-------------+ | TASK
| TEST |
+-------------------------------------+---------
The whole stack is split into two parts - HARNESS and TASK
HARNESS is task independent part - this will be the same regardless of
type of
test, and where possible platform/architecture independent.
TASK is any script/executable compatible with HARNESS interface.
==== HARNESS ====
HARNESS will be installed and set-up during provisioning phase (or as
packages
by developer in case of manual execution)
* test-controller - the heart of HARNESS - download, install, run, log and
report. test-controller has to be able running multiple tasks at the same
time and handle concurrent input from them.
* controller-lib - system/platform dependent part
* system dependent set-up
* system dependent routines - e.g. use yum/up2date or wget?
* server-adaptor - the only backend(LC) dependent module.
(beaker|local|...)
* there will be more adaptors allowed at the same time - e.g. one
to server, another for displaying results locally, and one to store
them
directly in DB. Local adapters can be attached/detached.
==== TASK ====
TASK will be downloaded and installed by HARNESS (or developer in case of
manual execution)
* TEST was described in general overview
* test-adaptor is adaptation layer conformant to HARNESS API. It is the
only
TEST dependent component (apart from the test itself.) It will
* install, setup, execute the script
* capture and analyse output of TEST and report back to HARNESS
* harness-lib provides language bindings to HARNESS API for
harness-aware test
scripts. It is the only test implementation language dependent component.
Library will provide functions for logging, file-uploading, streaming,
result-sumbitting, task-spawning, synchronizing, rebooting...
=== FOOTNOTES ===
==== System reboot handling ====
TASK should not be allowed to reboot the system, as this would affect all
concurrently runing tasks. It has to signal harness, which will suspend
reboot
untill it is safe. All the running TASKS has to be cooperative, or we would
have to wait untill the task finishes.
==== Meta-tasks ====
Task comprising of many subtasks and rules for executing them:
* Sequential|Parallel execution
* Continue when Any|At-most(N)|None tests fails
* Repeat(N) - run single task N times
Though this could be handled on Server-side, but it would not allow running
such tests locally.
==== Concurrency ====
For tasks wishing to spawn other tasks, it would be useful, if these were
registerred to harness.
=== IMPLEMENTATION ===
1. Interface between HARNESS and TASK
* Marian suggests using single named pipe with structured data
(xml-rpc? Tag-Length-Value encoded data?)
* most of the traffic from TASK to HARNESS will be simply forwarded by
test-controller to server-adaptor.
2. Implement server-adaptor for displaying to console/redirecting to a
file.
3. Implement basic functionality of test-controller && controller lib
* install, set-up, execute, capture and report,
* no concurrency, no spawning new tasks
4. Implement a test-adaptor for Perl's Test::Harness::TAP format
* It is simple enough for evaluation
5. Run some tests
6. Go back to 1, iterate...
--------------------------------------------------------------------------------
Heh, I spent few hours sorting ideas and writing this. Should cancel
SICK DAY
and report it as 1/2 day... ;-)
Looks like it could be worthy publishing on wiki. Any suggestions for
improvements, before I do so?
Cheers,
-- Marian
Show replies by thread