Hi Jaroslav,
----- Original Message -----
From: "Jaroslav Kortus" <jkortus(a)redhat.com>
To: "Beaker development" <beaker-devel(a)lists.fedorahosted.org>
Sent: Saturday, February 28, 2015 7:00:07 AM
Subject: [Beaker-devel] Integration Tests results
Hi there,
I've just executed full integration tests run with the following result
(patched develop
branch):
Ran 1345 tests in 6731.735s
FAILED (SKIP=3, errors=70, failures=55)
real 112m14.756s
user 101m11.856s
sys 7m42.248s
Quite some errors then. Much more than I expected :). Looking at the results
I have mixed
feelings.
That's a lot of errors. What distro are you using? We run our pre-merge tests on RHEL
6.6
and we get a near 100% pass all the time. I personally use Fedora 21 and I get a few
failures
mainly due to incompatibilities of third-party libraries between releases. If you have
docker installed,
you can try this approach of running your tests in a F21 container - it will install all
the dependencies
required/database setup, etc:
https://github.com/amitsaha/docker_files/tree/master/dev_workflow/beaker-...
Some selenium tests were failing (almost all in my case). When I ran the same
test alone
it passed with no problems.
Can you please paste the error messages? Perhaps fpaste it?
To get more data I ran the suite with "-sv" switch. Looking at the output
there are some
things that I'm not sure are errors or not.
For instance:
bkr.server.mail: ERROR: Exception thrown when trying to send mail
[...]
error: [Errno 111] Connection refused
bkr.server.tests.data_setup: DEBUG: Marked R:706 as complete with result Pass
Yeah, you can safely ignore this. It needs a local SMTP daemon running iirc and ignores
it
it if it doesn't find it.
or:
FAIL: test_all_tables_use_innodb
(bkr.inttest.server.test_model.SchemaSanityTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py",
line
55, in test_all_tables_use_innodb
table), 'InnoDB')
AssertionError: u'MyISAM' != 'InnoDB'
I've not created any new tables there (only added columns to existing
tables). So how can
this happen?
Sorry, not sure what you mean by that. Beaker uses the InnoDB storage engine, so
MariaDB/MySQL
should be configured appropriately [1].
[1]
https://beaker-project.org/dev/guide/getting-started.html
So the main question here is, are all the tests expected to pass on the
develop branch? Or
what are the usual results I can compare my patches to? Are there any other
requirements
besides the deps of integration tests RPM and
https://beaker-project.org/dev/guide/writing-a-patch.html#testing-your-pa...
On RHEL 6.6/ CentOS, they should all pass. On Fedora 20/21, you may have a few failures.
We have a script to setup everything needed for running the tests:
https://git.beaker-project.org/cgit/beaker/tree/Misc/beaker_test_setup.sh...
But, it needs an update. The part where it downloads the Selenium JAR is no longer
relevant.
Thank you,
Jaroslav.
PS: are the results saved somewhere or is stdout the only option?
"--debug-log" looks more
like the framework debugging than the actual tests' debug output...
I think stdout is the only option.
Hope that helps,
Amit.
--
Amit Saha
SED, Hosted & Shared Services
Red Hat, Inc.