Greetings,
I've been looking into integrating existing oz unittests with jenkins. It wasn't hard to have them run in jenkins as is, but I was looking give jenkins a better idea for pass/fail data over time. One way to do this was to convert the tests to use py.test (or unittest -- but that's a little crusty). Py.test can emit junit XML test output, which is jenkins friendly, and has much less stock test class/method bloat. I've modified the existing oz unittests to be py.test friendly.
Another annoyance with running tests in jenkins is that the nodes need to be manually setup with appropriate dependencies ahead of time. While this isn't hard, it's maintenance and one more thing to forget/fail when testing. The new test driver (runtests.sh) will setup a python virtualenv and install required dependencies there. This is intended to handle deps installation during unittest execution.
.gitignore | 6 + Makefile | 6 +- tests/dependencies.txt | 2 + tests/factory/run.sh | 5 - tests/factory/test_factory.py | 311 +++++++++++++++++++++-------------------- tests/runtests.sh | 90 ++++++++++++ tests/tdl/run.sh | 128 ----------------- tests/tdl/test.cfg | 135 ++++++++++++++++++ tests/tdl/test_tdl.py | 131 +++++++++++++++++- 9 files changed, 524 insertions(+), 290 deletions(-)
Comments appreciated.
Thanks, James
--- .gitignore | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-) create mode 100644 .gitignore
diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..7670203 --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +*.gz +*.pyc +*.sw[po] +*~ +tests/test_env +tests/test_results.xml
The runtests.sh script is responsible for ... * Satisfying test dependencies by creating a virtualenv * Running all applicable tests using py.test --- Makefile | 6 +- tests/dependencies.txt | 2 + tests/factory/run.sh | 5 -- tests/runtests.sh | 87 ++++++++++++++++++++++++++++++++ tests/tdl/run.sh | 128 ------------------------------------------------ 5 files changed, 92 insertions(+), 136 deletions(-) create mode 100644 tests/dependencies.txt delete mode 100755 tests/factory/run.sh create mode 100755 tests/runtests.sh delete mode 100755 tests/tdl/run.sh
diff --git a/Makefile b/Makefile index 0957518..d228378 100644 --- a/Makefile +++ b/Makefile @@ -28,11 +28,11 @@ man2html: @groff -mandoc man/oz-cleanup-cache.1 -T html > man/oz-cleanup-cache.html
unittests: - @cd tests/tdl ; ./run.sh - @cd tests/factory ; ./run.sh + @bash tests/runtests.sh +# @cd tests/factory ; ./run.sh
pylint: pylint --rcfile=pylint.conf oz oz-install oz-customize oz-cleanup-cache oz-generate-icicle
clean: - rm -rf MANIFEST build dist usr *~ oz.spec *.pyc oz/*~ oz/*.pyc examples/*~ oz/auto/*~ man/*~ docs/*~ man/*.html tests/tdl/*~ tests/factory/*~ + rm -rf MANIFEST build dist usr *~ oz.spec *.pyc oz/*~ oz/*.pyc examples/*~ oz/auto/*~ man/*~ docs/*~ man/*.html tests/tdl/*~ tests/factory/*~ tests/test_results.xml tests/test_env diff --git a/tests/dependencies.txt b/tests/dependencies.txt new file mode 100644 index 0000000..7f0a740 --- /dev/null +++ b/tests/dependencies.txt @@ -0,0 +1,2 @@ +pytest>=2.0.1 +libxml2dom diff --git a/tests/factory/run.sh b/tests/factory/run.sh deleted file mode 100755 index 0e00333..0000000 --- a/tests/factory/run.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/bash - -export PYTHONPATH=../..:$PYTHONPATH - -python test_factory.py diff --git a/tests/runtests.sh b/tests/runtests.sh new file mode 100755 index 0000000..961d9d4 --- /dev/null +++ b/tests/runtests.sh @@ -0,0 +1,87 @@ +#!/bin/bash +# Test driver responsible for satisfying test dependencies using virtualenv, +# and running available tests + +# Used for the name of the virtualenv +ENV_PATH='tests/test_env' + +# pip-python's requirements +REQ_FILE='tests/dependencies.txt' + +# this is to make sure that pip-python isn't going to touch anything but the +# virtualenv +PIP_REQUIRE_VIRTUALENV=1 + +# this keeps track of the python version since virtualenv will use the system +# python version by default +PYTHON_SITELIB=$(python -c "from distutils.sysconfig import get_python_lib; print get_python_lib(prefix='')") + +# Additional py.test command arguments +PYTEST_ARGS="--junitxml=tests/test_results.xml --ignore=${ENV_PATH}" + +# Display basic usage +usage() +{ + cat << EOF + usage: $0 + This script prepares a virtualenv for running the tests (if it doesn't + already exist) and runs unit tests in that virtualenv. + + OPTIONS: + -h Show this message + -v Enable Verbosity +EOF + exit 1 +} + +# Creates and prepares a virtualenv +create_env() +{ + echo "Virtualenv ($ENV_PATH) does not already exist." + echo "Creating virtualenv ... " + virtualenv $ENV_PATH + + echo "Installing packages (into $ENV_PATH) required for tests ... " + pip-python -E $ENV_PATH install -r $REQ_FILE + + echo "Adding oz to virtualenv site-packages ... " + pushd "$ENV_PATH/$PYTHON_SITELIB" + ln -sf ../../../../../oz + popd + + echo "Virtualenv created and prepared." +} + +# Process arguments +while getopts "hv" OPTION +do + case ${OPTION} in + v ) PYTEST_ARGS="${PYTEST_ARGS} --verbose" ;; + ?|h ) usage ;; + * ) usage ;; + esac +done + +# check for virtualenv +if [ ! -e $ENV_PATH ]; then + create_env +else + echo "Virtualenv ($ENV_PATH) detected." +fi + +# Activate the virtualenv +source "$ENV_PATH/bin/activate" + +# Run the tests +py.test ${PYTEST_ARGS} tests + +TEST_RESULT=$? + +# Deactivate the virtualenv +deactivate + +if [ $TEST_RESULT -ne 0 ]; then + exit 1 +else + exit 0 +fi diff --git a/tests/tdl/run.sh b/tests/tdl/run.sh deleted file mode 100755 index f44ce5c..0000000 --- a/tests/tdl/run.sh +++ /dev/null @@ -1,128 +0,0 @@ -#!/bin/bash - -export PYTHONPATH=../..:$PYTHONPATH - -if [ -z "$DEBUG" ]; then - DEBUG=0 -fi - -SUCCESS=0 -FAIL=0 - -success() { - echo "OK" - SUCCESS=$(( $SUCCESS + 1 )) -} - -failure() { - echo "FAIL" - FAIL=$(( $FAIL + 1 )) -} - -schema_test() { - schema="$1" - expectsuccess="$2" - - echo -n "Testing Schema $schema..." - - if [ ! -r "$1" ]; then - echo "FAIL: File not found" - FAIL=$(( $FAIL + 1 )) - return - fi - if [ $DEBUG -eq 0 ]; then - xmllint --noout --relaxng ../../docs/tdl.rng "$schema" >& /dev/null - else - xmllint --relaxng ../../docs/tdl.rng "$schema" - fi - - RET=$? - - if [ $RET -eq 0 -a "$expectsuccess" = "true" ] || [ $RET -ne 0 -a "$expectsuccess" != "true" ]; then - success - else - failure - fi -} - -python_test() { - parse="$1" - expectsuccess="$2" - - echo -n "Testing Parsing $1..." - if [ ! -r "$1" ]; then - echo "FAIL: File not found" - FAIL=$(( $FAIL + 1 )) - return - fi - - if [ $DEBUG -eq 0 ]; then - python test_tdl.py "$1" >& /dev/null - else - python test_tdl.py "$1" - fi - - RET=$? - - if [ $RET -eq 0 -a "$expectsuccess" = "true" ] || [ $RET -ne 0 -a "$expectsuccess" != "true" ]; then - success - else - failure - fi -} - -expect_success() { - schema_test "$1" "true" - python_test "$1" "true" -} - -expect_fail() { - schema_test "$1" "false" - python_test "$1" "false" -} - -expect_success test-01-simple-iso.tdl -expect_success test-02-simple-url.tdl -expect_fail test-03-empty-template.tdl -expect_fail test-04-no-os.tdl -expect_fail test-05-no-name.tdl -expect_success test-06-simple-iso-description.tdl -expect_success test-07-packages-no-package.tdl -expect_success test-08-repositories-no-repository.tdl -expect_fail test-09-os-invalid-arch.tdl -expect_fail test-10-os-invalid-install-type.tdl -expect_success test-11-description-packages-repositories.tdl -expect_fail test-12-os-no-name.tdl -expect_fail test-13-os-no-version.tdl -expect_fail test-14-os-no-arch.tdl -expect_fail test-15-os-no-install.tdl -expect_success test-16-signed-repository.tdl -expect_fail test-17-repo-invalid-signed.tdl -expect_success test-18-rootpw.tdl -expect_success test-19-key.tdl -expect_fail test-20-multiple-install.tdl -expect_fail test-21-missing-install-type.tdl -expect_success test-22-md5sum.tdl -expect_success test-23-sha1sum.tdl -expect_success test-24-sha256sum.tdl -expect_fail test-25-md5sum-and-sha1sum.tdl -expect_fail test-26-md5sum-and-sha256sum.tdl -expect_fail test-27-sha1sum-and-sha256sum.tdl -expect_fail test-28-package-no-name.tdl -expect_success test-29-files.tdl -expect_fail test-30-file-no-name.tdl -expect_success test-31-file-raw-type.tdl -expect_success test-32-file-base64-type.tdl -expect_fail test-33-file-invalid-type.tdl -expect_fail test-34-file-invalid-base64.tdl -expect_fail test-35-repository-no-name.tdl -expect_fail test-36-repository-no-url.tdl -expect_success test-37-command.tdl -expect_fail test-38-command-no-name.tdl -expect_success test-39-command-raw-type.tdl -expect_success test-40-command-base64-type.tdl -expect_fail test-41-command-bogus-base64.tdl -expect_fail test-42-command-bogus-type.tdl - -echo "SUCCESS: $SUCCESS, FAIL: $FAIL" -exit $FAIL
* test_tdl.py - redesigned to comply with py.test format * test_tdl.py - use libxml2 directly for xmllint validation * test.cfg - Expected test results no longer hard-coded, results are managed using ConfigParser-style test.cfg file (see [DEFAULT] for description of options) --- tests/tdl/test.cfg | 135 +++++++++++++++++++++++++++++++++++++++++++++++++ tests/tdl/test_tdl.py | 131 ++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 262 insertions(+), 4 deletions(-) create mode 100644 tests/tdl/test.cfg
diff --git a/tests/tdl/test.cfg b/tests/tdl/test.cfg new file mode 100644 index 0000000..f929f04 --- /dev/null +++ b/tests/tdl/test.cfg @@ -0,0 +1,135 @@ +# Main config file for oz tdl tests + +[DEFAULT] +# Unique test name, if no name is provided, the section name is used. +name = %(__name__)s + +# Expected overall test result. When set, individual test results are ignored. +# If no value is provided, individual results are used. +result = + +# Ozlib test result. If no value is provided, test is assumed pass. +ozlib_result = + +# xmllint schema test result. If no value is provided, test is assumed pass. +schema_result = + +# (optional) - Specify exact exception expected from test failure (default: +# oz.OzException.OzException). Note, the value used is passed to eval(). Use +# with caution!! +# ozlib_exception = + +# (optional) - Specify exact exception expected from test failure (default: +# Exception). Note, the value used is passed to eval(). Use with caution!! +# schema_exception = + +# All section names must match a .tdl file name +# Refer to [DEFAULT] for description of available options +[test-01-simple-iso.tdl] + +[test-02-simple-url.tdl] + +[test-03-empty-template.tdl] +result = fail + +[test-04-no-os.tdl] +result = fail + +[test-05-no-name.tdl] +result = fail + +[test-06-simple-iso-description.tdl] + +[test-07-packages-no-package.tdl] + +[test-08-repositories-no-repository.tdl] + +[test-09-os-invalid-arch.tdl] +result = fail + +[test-10-os-invalid-install-type.tdl] +result = fail + +[test-11-description-packages-repositories.tdl] + +[test-12-os-no-name.tdl] +result = fail + +[test-13-os-no-version.tdl] +result = fail + +[test-14-os-no-arch.tdl] +result = fail + +[test-15-os-no-install.tdl] +result = fail + +[test-16-signed-repository.tdl] + +[test-17-repo-invalid-signed.tdl] +result = fail + +[test-18-rootpw.tdl] + +[test-19-key.tdl] + +[test-20-multiple-install.tdl] +result = fail + +[test-21-missing-install-type.tdl] +result = fail + +[test-22-md5sum.tdl] + +[test-23-sha1sum.tdl] + +[test-24-sha256sum.tdl] + +[test-25-md5sum-and-sha1sum.tdl] +result = fail + +[test-26-md5sum-and-sha256sum.tdl] +result = fail + +[test-27-sha1sum-and-sha256sum.tdl] +result = fail + +[test-28-package-no-name.tdl] +result = fail + +[test-29-files.tdl] + +[test-30-file-no-name.tdl] +result = fail + +[test-31-file-raw-type.tdl] + +[test-32-file-base64-type.tdl] + +[test-33-file-invalid-type.tdl] +result = fail + +[test-34-file-invalid-base64.tdl] +result = fail +ozlib_exception = TypeError + +[test-35-repository-no-name.tdl] +result = fail + +[test-36-repository-no-url.tdl] +result = fail + +[test-37-command.tdl] + +[test-38-command-no-name.tdl] +result = fail + +[test-39-command-raw-type.tdl] +[test-40-command-base64-type.tdl] + +[test-41-command-bogus-base64.tdl] +result = fail +ozlib_exception = TypeError + +[test-42-command-bogus-type.tdl] +result = fail diff --git a/tests/tdl/test_tdl.py b/tests/tdl/test_tdl.py index 75ea3f8..a2e03c4 100644 --- a/tests/tdl/test_tdl.py +++ b/tests/tdl/test_tdl.py @@ -1,8 +1,131 @@ -import oz.TDL import sys +import os +import glob +import ConfigParser
-if len(sys.argv) != 2: - print "Usage: test_tdl.py <tdl>" +try: + import libxml2 +except ImportError: + print 'Unable to import libxml2. Is libxml2-python installed?' sys.exit(1)
-tdl = oz.TDL.TDL(open(sys.argv[1], 'r').read()) +try: + import py.test +except ImportError: + print 'Unable to import py.test. Is py.test installed?' + sys.exit(1) + +try: + import oz + import oz.TDL +except ImportError: + print 'Unable to import oz. Is oz installed?' + sys.exit(1) + +# Validate oz handling of tdl file +def validate_ozlib(tdl_file): + xmldata = open(tdl_file, 'r').read() + tdl = oz.TDL.TDL(xmldata) + +# Validate schema +def validate_schema(tdl_file): + + # Locate relaxng schema + rng_file = None + for tryme in ['../../docs/tdl.rng', + '../docs/tdl.rng', + 'docs/tdl.rng', + 'tdl.rng',]: + if os.path.isfile(tryme): + rng_file = tryme + break + + if rng_file is None: + raise Exception('RelaxNG schema file not found: tdl.rng') + + # Load relaxng schema + schema = open(rng_file, 'r').read() + rngp = libxml2.relaxNGNewMemParserCtxt(schema, len(schema)) + rngs = rngp.relaxNGParse() + + # Define callback for error handling + def error_cb(ctx, str): + print "%s: %s" % (ctx, str.strip()) + libxml2.registerErrorHandler(error_cb, tdl_file) + + # Attempt to validate + reader = libxml2.newTextReaderFilename(tdl_file) + #reader.SetParserProp(libxml2.PARSER_VALIDATE, 1) + reader.RelaxNGSetSchema(rngs) + ret = reader.Read() + while ret == 1: + ret = reader.Read() + + if ret != 0: + raise Exception('Error parsing the document: %s' % tdl_file) + if reader.IsValid() != 1: + raise Exception('Document failed to validate: %s' % tdl_file) + +# Test generator that iterates over all .tdl files +def test_all_tdls(): + + # Define a helper to expect an exception + def handle_exception(exc, func, *args): + with py.test.raises(exc): + func(*args) + + # Load test configuration file + test_path = os.path.join('tests', 'tdl') + cfg = ConfigParser.ConfigParser() + cfg.read(os.path.join(test_path, 'test.cfg')) + + # Sanity check to see if any tests are unaccounted for in the config file + tdl_files = glob.glob('tests/tdl/*.tdl') + for tdl_file in tdl_files: + # we want the file, not the full file path + tdl_file = os.path.basename(tdl_file) + if not cfg.has_section(tdl_file): + py.test.fail('Missing test.cfg section for: %s' % tdl_file) + + # Iterate over all expected good tdl files + for test_name in cfg.sections(): + + test_desc = cfg.get(test_name, 'name') + tdl_path = os.path.join(test_path, test_name) + + # load expected results + expected_overall = cfg.get(test_name, 'result', '') + expected_ozlib = cfg.get(test_name, 'ozlib_result', '') + expected_schema = cfg.get(test_name, 'schema_result', '') + + # load possible expected failure conditions + ozlib_exception = cfg.has_option(test_name, 'ozlib_exception') \ + and cfg.get(test_name, 'ozlib_exception') or oz.OzException.OzException + schema_exception = cfg.has_option(test_name, 'schema_exception') \ + and cfg.get(test_name, 'schema_exception') or Exception + + # Convert an exception string into an exception object + if isinstance(ozlib_exception, str): + try: + ozlib_exception = eval(ozlib_exception) + except NameError: + py.test.fail('Unknown exception: %s' % ozlib_exception) + if isinstance(schema_exception, str): + try: + schema_exception = eval(schema_exception) + except NameError: + py.test.fail('Unknown exception: %s' % schema_exception) + + # Run validate_ozlib test + if expected_overall in ['','pass'] and expected_ozlib in ['', 'pass']: + yield '%s (ozlib)' % test_desc, validate_ozlib, tdl_path + else: # expected FAIL + yield '%s (ozlib)' % test_desc, handle_exception, \ + ozlib_exception, validate_ozlib, tdl_path + + # Run validate_schema test + if expected_overall in ['','pass'] and expected_schema in ['', 'pass']: + yield '%s (schema)' % test_desc, validate_schema, tdl_path + else: # expected FAIL + yield '%s (schema)' % test_desc, handle_exception, \ + schema_exception, validate_schema, tdl_path
--- tests/factory/test_factory.py | 311 +++++++++++++++++++++-------------------- 1 files changed, 161 insertions(+), 150 deletions(-)
diff --git a/tests/factory/test_factory.py b/tests/factory/test_factory.py index 934277b..b05ce67 100644 --- a/tests/factory/test_factory.py +++ b/tests/factory/test_factory.py @@ -1,13 +1,21 @@ -import oz.TDL -import oz.GuestFactory import ConfigParser import StringIO import logging import os import sys
-success = 0 -fail = 0 +try: + import py.test +except ImportError: + print 'Unable to import py.test. Is py.test installed?' + sys.exit(1) + +try: + import oz.TDL + import oz.GuestFactory +except ImportError: + print 'Unable to import oz. Is oz installed?' + sys.exit(1)
def default_route(): route_file = "/proc/net/route" @@ -33,7 +41,7 @@ def default_route(): # doesn't really matter; it just has to have an IP address route = default_route()
-def test_distro(distro, version, arch, installtype): +def runtest(distro, version, arch, installtype): global route
tdlxml = """ @@ -63,153 +71,156 @@ def test_distro(distro, version, arch, installtype):
guest = oz.GuestFactory.guest_factory(tdl, config, None)
-def runtest(distro, version, arch, installtype, expect_success): - global success - global fail - - print "Testing %s-%s-%s-%s..." % (distro, version, arch, installtype), - try: - test_distro(distro, version, arch, installtype) - if expect_success: - success += 1 - print "OK" - else: - fail += 1 - print "FAIL" - except Exception, e: - if expect_success: - fail += 1 - print e - print "FAIL" - else: - success += 1 - print "OK" - -def expect_success(distro, version, arch, installtype): - return runtest(distro, version, arch, installtype, True) - -def expect_fail(distro, version, arch, installtype): - return runtest(distro, version, arch, installtype, False) - -# bad distro -expect_fail("foo", "1", "i386", "url") -# bad installtype -expect_fail("Fedora", "14", "i386", "dong") -# bad arch -expect_fail("Fedora", "14", "ia64", "iso") - -# FedoraCore -for version in ["1", "2", "3", "4", "5", "6"]: - for arch in ["i386", "x86_64"]: - for installtype in ["url", "iso"]: - expect_success("FedoraCore", version, arch, installtype) -# bad FedoraCore version -expect_fail("FedoraCore", "24", "x86_64", "iso") - -# Fedora -for version in ["7", "8", "9", "10", "11", "12", "13", "14", "15"]: - for arch in ["i386", "x86_64"]: - for installtype in ["url", "iso"]: - expect_success("Fedora", version, arch, installtype) -# bad Fedora version -expect_fail("Fedora", "24", "x86_64", "iso") - -# RHL -for version in ["7.0", "7.1", "7.2", "7.3", "8", "9"]: - expect_success("RHL", version, "i386", "url") -# bad RHL version -expect_fail("RHL", "10", "i386", "url") -# bad RHL arch -expect_fail("RHL", "9", "x86_64", "url") -# bad RHL installtype -expect_fail("RHL", "9", "x86_64", "iso") - -# RHEL-2.1 -for version in ["GOLD", "U2", "U3", "U4", "U5", "U6"]: - expect_success("RHEL-2.1", version, "i386", "url") -# bad RHEL-2.1 version -expect_fail("RHEL-2.1", "U7", "i386", "url") -# bad RHEL-2.1 arch -expect_fail("RHEL-2.1", "U6", "x86_64", "url") -# bad RHEL-2.1 installtype -expect_fail("RHEL-2.1", "U6", "i386", "iso") - -# RHEL-3 -for version in ["GOLD", "U1", "U2", "U3", "U4", "U5", "U6", "U7", "U8", "U9"]: - for arch in ["i386", "x86_64"]: - expect_success("RHEL-3", version, arch, "url") -# bad RHEL-3 version -expect_fail("RHEL-3", "U10", "x86_64", "url") -# invalid RHEL-3 installtype -expect_fail("RHEL-3", "U9", "x86_64", "iso") - -# RHEL-4/CentOS-4 -for distro in ["RHEL-4", "CentOS-4"]: - for version in ["GOLD", "U1", "U2", "U3", "U4", "U5", "U6", "U7", "U8", "U9"]: +def expect_success(*args): + runtest(*args) + +#def expect_fail(distro, version, arch, installtype): +# return runtest(distro, version, arch, installtype, False) + +# Define a helper to expect an exception +def exception_expected(exc, func, *args): + with py.test.raises(exc): + runtest(*args) + +def expect_fail(*args): + with py.test.raises(Exception): + runtest(*args) + +def test_all(): + + # bad distro + yield 'bad-distro', expect_fail, "foo", "1", "i386", "url" + # bad installtype + yield 'bad-installtype', expect_fail, "Fedora", "14", "i386", "dong" + # bad arch + yield 'bad-arch', expect_fail, "Fedora", "14", "ia64", "iso" + + # FedoraCore + for version in ["1", "2", "3", "4", "5", "6"]: for arch in ["i386", "x86_64"]: for installtype in ["url", "iso"]: - expect_success(distro, version, arch, installtype) -# bad RHEL-4 version -expect_fail("RHEL-4", "U10", "x86_64", "url") + yield 'FedoraCore-%s-%s-%s' % (version, arch, installtype), \ + expect_success, "FedoraCore", version, arch, installtype + # bad FedoraCore version + yield 'bad-FedoraCore-version', expect_fail, "FedoraCore", "24", "x86_64", "iso"
-# RHEL-5/CentOS-5 -for distro in ["RHEL-5", "CentOS-5"]: - for version in ["GOLD", "U1", "U2", "U3", "U4", "U5", "U6", "U7"]: + # Fedora + for version in ["7", "8", "9", "10", "11", "12", "13", "14", "15"]: for arch in ["i386", "x86_64"]: for installtype in ["url", "iso"]: - expect_success(distro, version, arch, installtype) -# bad RHEL-5 version -expect_fail("RHEL-5", "U10", "x86_64", "url") - -# RHEL-6 -for version in ["0", "1"]: - for arch in ["i386", "x86_64"]: - for installtype in ["url", "iso"]: - expect_success("RHEL-6", version, arch, installtype) -# bad RHEL-6 version -expect_fail("RHEL-6", "U10", "x86_64", "url") - -# Debian -for version in ["5", "6"]: - for arch in ["i386", "x86_64"]: - expect_success("Debian", version, arch, "iso") -# bad Debian version -expect_fail("Debian", "12", "i386", "iso") -# invalid Debian installtype -expect_fail("Debian", "6", "x86_64", "url") - -# Windows -expect_success("Windows", "2000", "i386", "iso") -for version in ["XP", "2003", "2008", "7"]: - for arch in ["i386", "x86_64"]: - expect_success("Windows", version, arch, "iso") -# bad Windows 2000 arch -expect_fail("Windows", "2000", "x86_64", "iso") -# bad Windows version -expect_fail("Windows", "1999", "x86_64", "iso") -# invalid Windows installtype -expect_fail("Windows", "2008", "x86_64", "url") - -# OpenSUSE -for version in ["11.0", "11.1", "11.2", "11.3", "11.4"]: - for arch in ["i386", "x86_64"]: - expect_success("OpenSUSE", version, arch, "iso") -# bad OpenSUSE version -expect_fail("OpenSUSE", "16", "x86_64", "iso") -# invalid OpenSUSE installtype -expect_fail("OpenSUSE", "11.4", "x86_64", "url") - -# Ubuntu -for version in ["6.06", "6.06.1", "6.06.2", "6.10", "7.04", "7.10", "8.04", - "8.04.1", "8.04.2", "8.04.3", "8.04.4", "8.10", "9.04", "9.10", - "10.04", "10.04.1", "10.04.2", "10.04.3", "10.10", "11.04"]: - for arch in ["i386", "x86_64"]: - expect_success("Ubuntu", version, arch, "iso") -# bad Ubuntu version -expect_fail("Ubuntu", "10.9", "i386", "iso") -# bad Ubuntu installtype -expect_fail("Ubuntu", "10.10", "i386", "url") - -print "SUCCESS: %d, FAIL: %d" % (success, fail) - -sys.exit(fail) + yield 'Fedora-%s-%s-%s' % (version, arch, installtype), \ + expect_success, "Fedora", version, arch, installtype + # bad Fedora version + yield 'bad-Fedora-version', expect_fail, "Fedora", "24", "x86_64", "iso" + + # RHL + for version in ["7.0", "7.1", "7.2", "7.3", "8", "9"]: + yield 'RHL-%s' % version, expect_success, "RHL", version, "i386", "url" + + # bad RHL version + yield 'bad-RHL-version', expect_fail, "RHL", "10", "i386", "url" + # bad RHL arch + yield 'bad-RHL-arch', expect_fail, "RHL", "9", "x86_64", "url" + # bad RHL installtype + yield 'bad-RHL-installtype', expect_fail, "RHL", "9", "x86_64", "iso" + + # RHEL-2.1 + for version in ["GOLD", "U2", "U3", "U4", "U5", "U6"]: + yield 'RHEL21-%s' % version, expect_success, "RHEL-2.1", version, "i386", "url" + + # bad RHEL-2.1 version + yield 'bad-RHEL21-version', expect_fail, "RHEL-2.1", "U7", "i386", "url" + # bad RHEL-2.1 arch + yield 'bad-RHEL21-arch', expect_fail, "RHEL-2.1", "U6", "x86_64", "url" + # bad RHEL-2.1 installtype + yield 'bad-RHEL21-installtype', expect_fail, "RHEL-2.1", "U6", "i386", "iso" + + # RHEL-3 + for version in ["GOLD", "U1", "U2", "U3", "U4", "U5", "U6", "U7", "U8", "U9"]: + for arch in ["i386", "x86_64"]: + yield 'RHEL-3-%s-%s' % (version, arch), \ + expect_success, "RHEL-3", version, arch, "url" + + # bad RHEL-3 version + yield 'bad-RHEL3-version', expect_fail, "RHEL-3", "U10", "x86_64", "url" + # invalid RHEL-3 installtype + yield 'bad-RHEL3-installtype', expect_fail, "RHEL-3", "U9", "x86_64", "iso" + + # RHEL-4/CentOS-4 + for distro in ["RHEL-4", "CentOS-4"]: + for version in ["GOLD", "U1", "U2", "U3", "U4", "U5", "U6", "U7", "U8", "U9"]: + for arch in ["i386", "x86_64"]: + for installtype in ["url", "iso"]: + yield '%s-%s-%s-%s' % (distro, version, arch, installtype), \ + expect_success, distro, version, arch, installtype + # bad RHEL-4 version + yield 'bad-RHEL4-version', expect_fail, "RHEL-4", "U10", "x86_64", "url" + + # RHEL-5/CentOS-5 + for distro in ["RHEL-5", "CentOS-5"]: + for version in ["GOLD", "U1", "U2", "U3", "U4", "U5", "U6", "U7"]: + for arch in ["i386", "x86_64"]: + for installtype in ["url", "iso"]: + yield '%s-%s-%s-%s' % (distro, version, arch, installtype), \ + expect_success, distro, version, arch, installtype + # bad RHEL-5 version + yield 'bad-RHEL5-version', expect_fail, "RHEL-5", "U10", "x86_64", "url" + + # RHEL-6 + for version in ["0", "1"]: + for arch in ["i386", "x86_64"]: + for installtype in ["url", "iso"]: + yield 'RHEL-6-%s-%s-%s' % (version, arch, installtype), \ + expect_success, "RHEL-6", version, arch, installtype + + # bad RHEL-6 version + yield 'bad-RHEL6-version', expect_fail, "RHEL-6", "U10", "x86_64", "url" + + # Debian + for version in ["5", "6"]: + for arch in ["i386", "x86_64"]: + yield 'Debian-%s-%s' % (version, arch), \ + expect_success, "Debian", version, arch, "iso" + + # bad Debian version + yield 'bad-debian-version', expect_fail, "Debian", "12", "i386", "iso" + # invalid Debian installtype + yield 'bad-debian-installtype', expect_fail, "Debian", "6", "x86_64", "url" + + # Windows + yield 'Windows-2000-i386-iso', expect_success, "Windows", "2000", "i386", "iso" + for version in ["XP", "2003", "2008", "7"]: + for arch in ["i386", "x86_64"]: + yield 'Windows-%s-%s' % (version, arch), \ + expect_success, "Windows", version, arch, "iso" + + # bad Windows 2000 arch + yield 'bad-windows-arch', expect_fail, "Windows", "2000", "x86_64", "iso" + # bad Windows version + yield 'bad-windows-version', expect_fail, "Windows", "1999", "x86_64", "iso" + # invalid Windows installtype + yield 'bad-windows-installtype', expect_fail, "Windows", "2008", "x86_64", "url" + + # OpenSUSE + for version in ["11.0", "11.1", "11.2", "11.3", "11.4"]: + for arch in ["i386", "x86_64"]: + yield 'OpenSUSE-%s-%s' % (version, arch), \ + expect_success, "OpenSUSE", version, arch, "iso" + + # bad OpenSUSE version + yield 'bad-OpenSUSE-version', expect_fail, "OpenSUSE", "16", "x86_64", "iso" + # invalid OpenSUSE installtype + yield 'bad-OpenSUSE-installtype', expect_fail, "OpenSUSE", "11.4", "x86_64", "url" + + # Ubuntu + for version in ["6.06", "6.06.1", "6.06.2", "6.10", "7.04", "7.10", "8.04", + "8.04.1", "8.04.2", "8.04.3", "8.04.4", "8.10", "9.04", "9.10", + "10.04", "10.04.1", "10.04.2", "10.04.3", "10.10", "11.04"]: + for arch in ["i386", "x86_64"]: + yield 'Ubuntu-%s-%s' % (version, arch), \ + expect_success, "Ubuntu", version, arch, "iso" + + # bad Ubuntu version + yield 'bad-Ubuntu-version', expect_fail, "Ubuntu", "10.9", "i386", "iso" + # bad Ubuntu installtype + yield 'bad-Ubuntu-installtype', expect_fail, "Ubuntu", "10.10", "i386", "url"
--- tests/runtests.sh | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/tests/runtests.sh b/tests/runtests.sh index 961d9d4..8600f72 100755 --- a/tests/runtests.sh +++ b/tests/runtests.sh @@ -12,6 +12,9 @@ REQ_FILE='tests/dependencies.txt' # virtualenv PIP_REQUIRE_VIRTUALENV=1
+# Where to look for tests (default: tests) +RUNTESTS=${@:-tests} + # this keeps track of the python version since virtualenv will use the system # python version by default PYTHON_SITELIB=$(python -c "from distutils.sysconfig import get_python_lib; print get_python_lib(prefix='')") @@ -23,7 +26,7 @@ PYTEST_ARGS="--junitxml=tests/test_results.xml --ignore=${ENV_PATH}" usage() { cat << EOF - usage: $0 + usage: $0 [tests] This script prepares a virtualenv for running the tests (if it doesn't already exist) and runs unit tests in that virtualenv.
@@ -73,7 +76,7 @@ fi source "$ENV_PATH/bin/activate"
# Run the tests -py.test ${PYTEST_ARGS} tests +py.test ${PYTEST_ARGS} ${RUNTESTS}
TEST_RESULT=$?
On 08/31/11 - 10:49:36AM, James Laska wrote:
Greetings,
I've been looking into integrating existing oz unittests with jenkins. It wasn't hard to have them run in jenkins as is, but I was looking give jenkins a better idea for pass/fail data over time. One way to do this was to convert the tests to use py.test (or unittest -- but that's a little crusty). Py.test can emit junit XML test output, which is jenkins friendly, and has much less stock test class/method bloat. I've modified the existing oz unittests to be py.test friendly.
Another annoyance with running tests in jenkins is that the nodes need to be manually setup with appropriate dependencies ahead of time. While this isn't hard, it's maintenance and one more thing to forget/fail when testing. The new test driver (runtests.sh) will setup a python virtualenv and install required dependencies there. This is intended to handle deps installation during unittest execution.
.gitignore | 6 + Makefile | 6 +- tests/dependencies.txt | 2 + tests/factory/run.sh | 5 - tests/factory/test_factory.py | 311 +++++++++++++++++++++-------------------- tests/runtests.sh | 90 ++++++++++++ tests/tdl/run.sh | 128 ----------------- tests/tdl/test.cfg | 135 ++++++++++++++++++ tests/tdl/test_tdl.py | 131 +++++++++++++++++- 9 files changed, 524 insertions(+), 290 deletions(-)
Arg. I know I said I was willing to entertain these patches, and I was, but I have to say that I am not a fan. It turns what is simple bash and python unittests into a morass of config files, scripts, and magic. Plus it adds 3 new dependencies to run the unittests, which is an additional barrier to entry for new contributors.
So there are 3 main problems that you see with the current unittests (the 2 listed above plus one you mentioned on IRC): 1) It is a sort of odd combination of bash and python, and you need to modify the code every time you want to add a new test 2) You want JUnit XML output to feed to jenkins 3) You want to have some sort of automatic dependency resolution for jenkins.
I think we can easily handle 1) by re-writing the small portions of this that are in bash in python, and make it smarter. For instance, take what you've done for moving the tests in their own directory, and just have the python code iterate over those tests.
For 2), after seeing the output of these patches, it seems to me that it should be easy enough to have the existing python code generate JUnit XML as output. Maybe we have the code take a flag (--junitxml=output/path), and then we can build it up ourselves (it is, after all, simple XML).
The third problem you have is a little more difficult. I want to keep running the tests as simple as possible, but you want automatic dependency stuff. I wonder if we can add a new "virtualenv" target to the Makefile, which only jenkins will call. Then it could call out to the tests as re-written above.
What do you think about that proposal?
On Thu, 2011-09-01 at 09:23 -0400, Chris Lalancette wrote:
On 08/31/11 - 10:49:36AM, James Laska wrote:
Greetings,
I've been looking into integrating existing oz unittests with jenkins. It wasn't hard to have them run in jenkins as is, but I was looking give jenkins a better idea for pass/fail data over time. One way to do this was to convert the tests to use py.test (or unittest -- but that's a little crusty). Py.test can emit junit XML test output, which is jenkins friendly, and has much less stock test class/method bloat. I've modified the existing oz unittests to be py.test friendly.
Another annoyance with running tests in jenkins is that the nodes need to be manually setup with appropriate dependencies ahead of time. While this isn't hard, it's maintenance and one more thing to forget/fail when testing. The new test driver (runtests.sh) will setup a python virtualenv and install required dependencies there. This is intended to handle deps installation during unittest execution.
.gitignore | 6 + Makefile | 6 +- tests/dependencies.txt | 2 + tests/factory/run.sh | 5 - tests/factory/test_factory.py | 311 +++++++++++++++++++++-------------------- tests/runtests.sh | 90 ++++++++++++ tests/tdl/run.sh | 128 ----------------- tests/tdl/test.cfg | 135 ++++++++++++++++++ tests/tdl/test_tdl.py | 131 +++++++++++++++++- 9 files changed, 524 insertions(+), 290 deletions(-)
Arg. I know I said I was willing to entertain these patches, and I was, but I have to say that I am not a fan. It turns what is simple bash and python unittests into a morass of config files, scripts, and magic. Plus it adds 3 new dependencies to run the unittests, which is an additional barrier to entry for new contributors.
What are the new dependencies, python-virtualenv and py.test?
I'm not a big fan of writing tests in bash, it's easy to write, and easier to write wrong and hard to debug. Plus there are no libs to rely on, you have to build the custom scaffolding. But that's just a personal preference.
I could have gone with unittest, which is built-in to python. But I've done it plenty of times and it requires too much scaffolding to do basic tasks. I liked py.test since you just write python scripts as your tests ... but it does add a new dependency.
So there are 3 main problems that you see with the current unittests (the 2 listed above plus one you mentioned on IRC):
- It is a sort of odd combination of bash and python, and you need to modify the code every time you want to add a new test
The current proposed patchset requires modifying a config file when adding new tests. I didn't consider that code, more config, but it does require a commit of sorts whenever adding/modifying tests. A previous version of the patchset required no modification at all, but offered less flexibility. Tests were split into two directories, expected_pass/ and expected_fail/ I can return to this if desired. I liked the current .cfg approach since it allowed for customizing the exact failure case, and I tried to self-document the use of options in the config file. If that's overkill, I can definitely return to the simpler directory-based approach.
- You want JUnit XML output to feed to jenkins
Yeah, I think that'd be helpful all around (unless the tests always pass) :)
- You want to have some sort of automatic dependency resolution for jenkins.
That'd be nice ... it's also for folks running 'make unittests' from a git checkout who don't have all the runtime/test dependencies installed. Rather, the later is how I've used it in the past.
I think we can easily handle 1) by re-writing the small portions of this that are in bash in python, and make it smarter. For instance, take what you've done for moving the tests in their own directory, and just have the python code iterate over those tests.
In the current approach, I left the .tdl test files where they were, but used a .cfg to define expected outcomes. Would it be easier to remove the .cfg and just have tests exist in expected_fail/ expected_pass/ ?
For 2), after seeing the output of these patches, it seems to me that it should be easy enough to have the existing python code generate JUnit XML as output. Maybe we have the code take a flag (--junitxml=output/path), and then we can build it up ourselves (it is, after all, simple XML).
My preference would be to avoid custom-built XML generation (that just seems prone to rot) and let the python framework (py.test) do that for us. I figured that would be less to maintain in the long-run. Let me know what you think.
The third problem you have is a little more difficult. I want to keep running the tests as simple as possible, but you want automatic dependency stuff. I wonder if we can add a new "virtualenv" target to the Makefile, which only jenkins will call. Then it could call out to the tests as re-written above.
Oh, I see ... so you'd be required to install the deps manually when running 'make unittests'. However, if testing with jenkins, you could do 'make virtualenv unittests' which would alleviate manual dependency installation? That's a neat idea, I could set that up.
What do you think about that proposal?
Thanks for your feedback! Whatever resolves the 3 problems you highlighted without introducing too much additional maintenance. I can get busy re-working based on your feedback.
Thanks, James
On 09/01/11 - 09:48:42AM, James Laska wrote:
Arg. I know I said I was willing to entertain these patches, and I was, but I have to say that I am not a fan. It turns what is simple bash and python unittests into a morass of config files, scripts, and magic. Plus it adds 3 new dependencies to run the unittests, which is an additional barrier to entry for new contributors.
What are the new dependencies, python-virtualenv and py.test?
And python-pip.
I'm not a big fan of writing tests in bash, it's easy to write, and easier to write wrong and hard to debug. Plus there are no libs to rely on, you have to build the custom scaffolding. But that's just a personal preference.
Yeah, I'm with you here. bash was just easy for me at the time, but I totally understand converting to python. Heck, the original version of Oz (before it was a full project) was all written in bash, and it was a mess :).
I could have gone with unittest, which is built-in to python. But I've done it plenty of times and it requires too much scaffolding to do basic tasks. I liked py.test since you just write python scripts as your tests ... but it does add a new dependency.
So there are 3 main problems that you see with the current unittests (the 2 listed above plus one you mentioned on IRC):
- It is a sort of odd combination of bash and python, and you need to modify the code every time you want to add a new test
The current proposed patchset requires modifying a config file when adding new tests. I didn't consider that code, more config, but it does require a commit of sorts whenever adding/modifying tests. A previous version of the patchset required no modification at all, but offered less flexibility. Tests were split into two directories, expected_pass/ and expected_fail/ I can return to this if desired. I liked the current .cfg approach since it allowed for customizing the exact failure case, and I tried to self-document the use of options in the config file. If that's overkill, I can definitely return to the simpler directory-based approach.
OK, so you don't care too much about modifying the code. I also don't care; you are going to have to commit a patch to add the new test anyway, so adding one more line to the test controller (run_test or whatever) doesn't seem like a huge deal. So the big point on 1) is converting to python. To that end, I've come up with the attached patch; let me know what you think.
- You want JUnit XML output to feed to jenkins
Yeah, I think that'd be helpful all around (unless the tests always pass) :)
- You want to have some sort of automatic dependency resolution for jenkins.
That'd be nice ... it's also for folks running 'make unittests' from a git checkout who don't have all the runtime/test dependencies installed. Rather, the later is how I've used it in the past.
I think we can easily handle 1) by re-writing the small portions of this that are in bash in python, and make it smarter. For instance, take what you've done for moving the tests in their own directory, and just have the python code iterate over those tests.
In the current approach, I left the .tdl test files where they were, but used a .cfg to define expected outcomes. Would it be easier to remove the .cfg and just have tests exist in expected_fail/ expected_pass/ ?
For 2), after seeing the output of these patches, it seems to me that it should be easy enough to have the existing python code generate JUnit XML as output. Maybe we have the code take a flag (--junitxml=output/path), and then we can build it up ourselves (it is, after all, simple XML).
My preference would be to avoid custom-built XML generation (that just seems prone to rot) and let the python framework (py.test) do that for us. I figured that would be less to maintain in the long-run. Let me know what you think.
The third problem you have is a little more difficult. I want to keep running the tests as simple as possible, but you want automatic dependency stuff. I wonder if we can add a new "virtualenv" target to the Makefile, which only jenkins will call. Then it could call out to the tests as re-written above.
Oh, I see ... so you'd be required to install the deps manually when running 'make unittests'. However, if testing with jenkins, you could do 'make virtualenv unittests' which would alleviate manual dependency installation? That's a neat idea, I could set that up.
Yeah, if you could look at doing that, that would at least address this part of the problem.
What do you think about that proposal?
Thanks for your feedback! Whatever resolves the 3 problems you highlighted without introducing too much additional maintenance. I can get busy re-working based on your feedback.
So with my proposed patch we've addressed 1) by converting everything to python. If you can work on 3) to add a virtualenv Makefile target, that would be great. That leaves us with 2); I'm totally with you on avoiding custom code where we can. I think it will require a bit more research, but if we can find something that can generate the junit XML (and only the junit XML), that might be workable.
On Fri, 2011-09-02 at 10:04 -0400, Chris Lalancette wrote:
On 09/01/11 - 09:48:42AM, James Laska wrote:
Arg. I know I said I was willing to entertain these patches, and I was, but I have to say that I am not a fan. It turns what is simple bash and python unittests into a morass of config files, scripts, and magic. Plus it adds 3 new dependencies to run the unittests, which is an additional barrier to entry for new contributors.
What are the new dependencies, python-virtualenv and py.test?
And python-pip.
Oh I forgot, the only hard requirement is python-virtualenv. Since anything else is handled magically by virtualenv. Using the nice idea of an optional virtualenv, those other requirements would be needed.
I'm not a big fan of writing tests in bash, it's easy to write, and easier to write wrong and hard to debug. Plus there are no libs to rely on, you have to build the custom scaffolding. But that's just a personal preference.
Yeah, I'm with you here. bash was just easy for me at the time, but I totally understand converting to python. Heck, the original version of Oz (before it was a full project) was all written in bash, and it was a mess :).
Haha, I thought after I wrote this that it might come across that your bash was bad. Apologies, that was not at all the case. It was actually pretty easy to read and you could tell it was done by someone with development experience.
I could have gone with unittest, which is built-in to python. But I've done it plenty of times and it requires too much scaffolding to do basic tasks. I liked py.test since you just write python scripts as your tests ... but it does add a new dependency.
So there are 3 main problems that you see with the current unittests (the 2 listed above plus one you mentioned on IRC):
- It is a sort of odd combination of bash and python, and you need to modify the code every time you want to add a new test
The current proposed patchset requires modifying a config file when adding new tests. I didn't consider that code, more config, but it does require a commit of sorts whenever adding/modifying tests. A previous version of the patchset required no modification at all, but offered less flexibility. Tests were split into two directories, expected_pass/ and expected_fail/ I can return to this if desired. I liked the current .cfg approach since it allowed for customizing the exact failure case, and I tried to self-document the use of options in the config file. If that's overkill, I can definitely return to the simpler directory-based approach.
OK, so you don't care too much about modifying the code. I also don't care; you are going to have to commit a patch to add the new test anyway, so adding one more line to the test controller (run_test or whatever) doesn't seem like a huge deal.
Yeah, true.
So the big point on 1) is converting to python. To that end, I've come up with the attached patch; let me know what you think.
Nice, thank you! I'll play with this shortly and get some feedback.
So with my proposed patch we've addressed 1) by converting everything to python. If you can work on 3) to add a virtualenv Makefile target, that would be great. That leaves us with 2); I'm totally with you on avoiding custom code where we can. I think it will require a bit more research, but if we can find something that can generate the junit XML (and only the junit XML), that might be workable.
I found these two projects that might work ... https://github.com/danielfm/unittest-xml-reporting https://launchpad.net/pyjunitxml
I tested unittest-xml-reporting and that seems to work. I haven't tried pyjunitxml. Either would need to be packaged and included in Fedora. I'll be glad to work that.
Thanks, James
On Thu, 2011-09-01 at 09:23 -0400, Chris Lalancette wrote:
On 08/31/11 - 10:49:36AM, James Laska wrote:
Greetings,
I've been looking into integrating existing oz unittests with jenkins. It wasn't hard to have them run in jenkins as is, but I was looking give jenkins a better idea for pass/fail data over time. One way to do this was to convert the tests to use py.test (or unittest -- but that's a little crusty). Py.test can emit junit XML test output, which is jenkins friendly, and has much less stock test class/method bloat. I've modified the existing oz unittests to be py.test friendly.
Another annoyance with running tests in jenkins is that the nodes need to be manually setup with appropriate dependencies ahead of time. While this isn't hard, it's maintenance and one more thing to forget/fail when testing. The new test driver (runtests.sh) will setup a python virtualenv and install required dependencies there. This is intended to handle deps installation during unittest execution.
.gitignore | 6 + Makefile | 6 +- tests/dependencies.txt | 2 + tests/factory/run.sh | 5 - tests/factory/test_factory.py | 311 +++++++++++++++++++++-------------------- tests/runtests.sh | 90 ++++++++++++ tests/tdl/run.sh | 128 ----------------- tests/tdl/test.cfg | 135 ++++++++++++++++++ tests/tdl/test_tdl.py | 131 +++++++++++++++++- 9 files changed, 524 insertions(+), 290 deletions(-)
Arg. I know I said I was willing to entertain these patches, and I was, but I have to say that I am not a fan. It turns what is simple bash and python unittests into a morass of config files, scripts, and magic. Plus it adds 3 new dependencies to run the unittests, which is an additional barrier to entry for new contributors.
FWIW, I like what OpenStack's glance has done with their unit tests using virtualenv and nose. Maybe look at that?
https://github.com/openstack/glance
Cheers, Mark.
On Fri, 2011-09-02 at 07:12 +0100, Mark McLoughlin wrote:
On Thu, 2011-09-01 at 09:23 -0400, Chris Lalancette wrote:
On 08/31/11 - 10:49:36AM, James Laska wrote:
Greetings,
I've been looking into integrating existing oz unittests with jenkins. It wasn't hard to have them run in jenkins as is, but I was looking give jenkins a better idea for pass/fail data over time. One way to do this was to convert the tests to use py.test (or unittest -- but that's a little crusty). Py.test can emit junit XML test output, which is jenkins friendly, and has much less stock test class/method bloat. I've modified the existing oz unittests to be py.test friendly.
Another annoyance with running tests in jenkins is that the nodes need to be manually setup with appropriate dependencies ahead of time. While this isn't hard, it's maintenance and one more thing to forget/fail when testing. The new test driver (runtests.sh) will setup a python virtualenv and install required dependencies there. This is intended to handle deps installation during unittest execution.
.gitignore | 6 + Makefile | 6 +- tests/dependencies.txt | 2 + tests/factory/run.sh | 5 - tests/factory/test_factory.py | 311 +++++++++++++++++++++-------------------- tests/runtests.sh | 90 ++++++++++++ tests/tdl/run.sh | 128 ----------------- tests/tdl/test.cfg | 135 ++++++++++++++++++ tests/tdl/test_tdl.py | 131 +++++++++++++++++- 9 files changed, 524 insertions(+), 290 deletions(-)
Arg. I know I said I was willing to entertain these patches, and I was, but I have to say that I am not a fan. It turns what is simple bash and python unittests into a morass of config files, scripts, and magic. Plus it adds 3 new dependencies to run the unittests, which is an additional barrier to entry for new contributors.
FWIW, I like what OpenStack's glance has done with their unit tests using virtualenv and nose. Maybe look at that?
Thanks for the link Mark. I like their run_tests.sh driver in that it offers a few additional arguments for managing the virtualenv (--clean, --force, --no-virtual-env etc...). This might be helpful for Chris' suggestion of optionally using virtualenv.
They still use a unittest-style tests. I don't mind that structure since it is well documented, it's just extra scaffolding. All tests must be importable modules, subclass unittest.TestCase, use unusual (but documented) assert*() methods for doing normal python things (like comparisons and asserts) and rely on alphanum sorting to determine test execution order (which may change as tests mature).
A comparison between nose and py.test ... https://www.ibm.com/developerworks/aix/library/au-python_test/
Another comparison ... http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytest-t...
Using nose would still require an additional dependency (python-nose). I guess it really comes down to preference on what folks are comfortable with maintaining/using. I'm happy to adjust to use unittest or nose if desired. My current use of py.test was in part because of built-in support for --junit output (without requiring packaging another testrunner like HTMLTestRunner.py [1] or unittest-xml-reporting [2] or pyjunitxml [3]), getting something going faster using the existing code Chris developed and of course ... to explore/learn something new.
/me standing by
Thanks, James
[1] http://tungwaiyip.info/software/HTMLTestRunner.html [2] https://github.com/danielfm/unittest-xml-reporting [3] https://launchpad.net/pyjunitxml
aeolus-devel@lists.fedorahosted.org
-
Chris Lalancette -
James Laska -
Mark McLoughlin