Hi there,
I've just executed full integration tests run with the following result (patched develop branch): Ran 1345 tests in 6731.735s
FAILED (SKIP=3, errors=70, failures=55)
real 112m14.756s user 101m11.856s sys 7m42.248s
Quite some errors then. Much more than I expected :). Looking at the results I have mixed feelings.
Some selenium tests were failing (almost all in my case). When I ran the same test alone it passed with no problems.
To get more data I ran the suite with "-sv" switch. Looking at the output there are some things that I'm not sure are errors or not.
For instance: bkr.server.mail: ERROR: Exception thrown when trying to send mail [...] error: [Errno 111] Connection refused bkr.server.tests.data_setup: DEBUG: Marked R:706 as complete with result Pass
or: FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line 55, in test_all_tables_use_innodb table), 'InnoDB') AssertionError: u'MyISAM' != 'InnoDB'
I've not created any new tables there (only added columns to existing tables). So how can this happen?
So the main question here is, are all the tests expected to pass on the develop branch? Or what are the usual results I can compare my patches to? Are there any other requirements besides the deps of integration tests RPM and https://beaker-project.org/dev/guide/writing-a-patch.html#testing-your-patch...
Thank you, Jaroslav.
PS: are the results saved somewhere or is stdout the only option? "--debug-log" looks more like the framework debugging than the actual tests' debug output...
Hi Jaroslav,
----- Original Message -----
From: "Jaroslav Kortus" jkortus@redhat.com To: "Beaker development" beaker-devel@lists.fedorahosted.org Sent: Saturday, February 28, 2015 7:00:07 AM Subject: [Beaker-devel] Integration Tests results
Hi there,
I've just executed full integration tests run with the following result (patched develop branch): Ran 1345 tests in 6731.735s
FAILED (SKIP=3, errors=70, failures=55)
real 112m14.756s user 101m11.856s sys 7m42.248s
Quite some errors then. Much more than I expected :). Looking at the results I have mixed feelings.
That's a lot of errors. What distro are you using? We run our pre-merge tests on RHEL 6.6 and we get a near 100% pass all the time. I personally use Fedora 21 and I get a few failures mainly due to incompatibilities of third-party libraries between releases. If you have docker installed, you can try this approach of running your tests in a F21 container - it will install all the dependencies required/database setup, etc:
https://github.com/amitsaha/docker_files/tree/master/dev_workflow/beaker-tes...
Some selenium tests were failing (almost all in my case). When I ran the same test alone it passed with no problems.
Can you please paste the error messages? Perhaps fpaste it?
To get more data I ran the suite with "-sv" switch. Looking at the output there are some things that I'm not sure are errors or not.
For instance: bkr.server.mail: ERROR: Exception thrown when trying to send mail [...] error: [Errno 111] Connection refused bkr.server.tests.data_setup: DEBUG: Marked R:706 as complete with result Pass
Yeah, you can safely ignore this. It needs a local SMTP daemon running iirc and ignores it it if it doesn't find it.
or: FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest)
Traceback (most recent call last): File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line 55, in test_all_tables_use_innodb table), 'InnoDB') AssertionError: u'MyISAM' != 'InnoDB'
I've not created any new tables there (only added columns to existing tables). So how can this happen?
Sorry, not sure what you mean by that. Beaker uses the InnoDB storage engine, so MariaDB/MySQL should be configured appropriately [1].
[1] https://beaker-project.org/dev/guide/getting-started.html
So the main question here is, are all the tests expected to pass on the develop branch? Or what are the usual results I can compare my patches to? Are there any other requirements besides the deps of integration tests RPM and https://beaker-project.org/dev/guide/writing-a-patch.html#testing-your-patch...
On RHEL 6.6/ CentOS, they should all pass. On Fedora 20/21, you may have a few failures. We have a script to setup everything needed for running the tests: https://git.beaker-project.org/cgit/beaker/tree/Misc/beaker_test_setup.sh?h=...
But, it needs an update. The part where it downloads the Selenium JAR is no longer relevant.
Thank you, Jaroslav.
PS: are the results saved somewhere or is stdout the only option? "--debug-log" looks more like the framework debugging than the actual tests' debug output...
I think stdout is the only option.
Hope that helps, Amit.
Excerpts from Amit Saha's message of 2015-03-02 10:42 +10:00:
----- Original Message -----
From: "Jaroslav Kortus" jkortus@redhat.com To: "Beaker development" beaker-devel@lists.fedorahosted.org Sent: Saturday, February 28, 2015 7:00:07 AM Subject: [Beaker-devel] Integration Tests results
or: FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest)
Traceback (most recent call last): File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line 55, in test_all_tables_use_innodb table), 'InnoDB') AssertionError: u'MyISAM' != 'InnoDB'
I've not created any new tables there (only added columns to existing tables). So how can this happen?
Sorry, not sure what you mean by that. Beaker uses the InnoDB storage engine, so MariaDB/MySQL should be configured appropriately [1].
[1] https://beaker-project.org/dev/guide/getting-started.html
This is a strange failure. You don't actually need to set default-storage-engine=INNODB, that's just a convenience in case you create a table by hand during development. Beaker always explicitly requests InnoDB as the storage engine when it creates all tables. That test case is just to catch situations where we add a new table and forget to set mysql_engine='InnoDB' in the table definition.
If you are seeing it fail (and you haven't added any tables) then it means MySQL is silently ignoring the requested storage engine type. This might happen if you have somehow got a MySQL server without the InnoDB engine compiled in? The mysql-server package in RHEL6 works out of the box, are you using something different?
PS: are the results saved somewhere or is stdout the only option? "--debug-log" looks more like the framework debugging than the actual tests' debug output...
I think stdout is the only option.
Nose doesn't have any options to save the test output anywhere. There might be plugins which can produce more readable output if you have a large number of failures.
The --debug and --debug-log options refer to nose's own debugging which is not useful.
Bear in mind that there is two sources of messages in Beaker's tests: stdout, which has the output of all inferior processes (this includes the access logs and Beaker debug logs produced by the server under test), and the log subsystem, which has debug logs produced by the test cases *and* the Beaker code invoked by the test cases directly.
Both stdout and logging are captured by nose and both are displayed (non-interleaved) on failures. There are two separate options to disable nose's capturing: -s (--nocapture) disables stdout capturing, --nologcapture disables log capturing. But nose already displays both of those on failures so there is not really any reason to disable the capturing, unless you suspect that some setup in a successful test case is causing a failure in a subsequent test case.
Basically if you just pastebin the entire output of the nose run we should be able to see what is going wrong.
On 03/02/2015 11:06 AM, Dan Callaghan wrote:
Excerpts from Amit Saha's message of 2015-03-02 10:42 +10:00:
----- Original Message -----
From: "Jaroslav Kortus" jkortus@redhat.com To: "Beaker development" beaker-devel@lists.fedorahosted.org Sent: Saturday, February 28, 2015 7:00:07 AM Subject: [Beaker-devel] Integration Tests results
or: FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest)
Traceback (most recent call last): File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line 55, in test_all_tables_use_innodb table), 'InnoDB') AssertionError: u'MyISAM' != 'InnoDB'
I've not created any new tables there (only added columns to existing tables). So how can this happen?
Sorry, not sure what you mean by that. Beaker uses the InnoDB storage engine, so MariaDB/MySQL should be configured appropriately [1].
[1] https://beaker-project.org/dev/guide/getting-started.html
This is a strange failure. You don't actually need to set default-storage-engine=INNODB, that's just a convenience in case you create a table by hand during development. Beaker always explicitly requests InnoDB as the storage engine when it creates all tables. That test case is just to catch situations where we add a new table and forget to set mysql_engine='InnoDB' in the table definition.
If you are seeing it fail (and you haven't added any tables) then it means MySQL is silently ignoring the requested storage engine type. This might happen if you have somehow got a MySQL server without the InnoDB engine compiled in? The mysql-server package in RHEL6 works out of the box, are you using something different?
PS: are the results saved somewhere or is stdout the only option? "--debug-log" looks more like the framework debugging than the actual tests' debug output...
I think stdout is the only option.
Nose doesn't have any options to save the test output anywhere. There might be plugins which can produce more readable output if you have a large number of failures.
IIRC, nose-test-progressive is pretty good for that, since you get feedback on the failures as they happen.
Cheers, Nick.
Hi Amit,
thanks for the reaction.
On 03/02/2015 01:42 AM, Amit Saha wrote:
Hi Jaroslav,
----- Original Message -----
From: "Jaroslav Kortus" jkortus@redhat.com To: "Beaker development" beaker-devel@lists.fedorahosted.org Sent: Saturday, February 28, 2015 7:00:07 AM Subject: [Beaker-devel] Integration Tests results
Hi there,
I've just executed full integration tests run with the following result (patched develop branch): Ran 1345 tests in 6731.735s
FAILED (SKIP=3, errors=70, failures=55)
real 112m14.756s user 101m11.856s sys 7m42.248s
Quite some errors then. Much more than I expected :). Looking at the results I have mixed feelings.
That's a lot of errors. What distro are you using? We run our pre-merge tests on RHEL 6.6 and we get a near 100% pass all the time. I personally use Fedora 21 and I get a few failures mainly due to incompatibilities of third-party libraries between releases. If you have docker installed, you can try this approach of running your tests in a F21 container - it will install all the dependencies required/database setup, etc:
https://github.com/amitsaha/docker_files/tree/master/dev_workflow/beaker-tes...
I was running them on 6.5GA, have upgraded now to 6.6, but the issues remain. However, the clean develop branch does not have that many errors by far, so I suspect I patched something that looks innocent but causes mayhem in the tests :). Will dig into it deeper to find it out...
Some selenium tests were failing (almost all in my case). When I ran the same test alone it passed with no problems.
Can you please paste the error messages? Perhaps fpaste it?
That was my problem - there are no error messages in the normal run-tests.sh run. If I run the test separately it succeeds.
FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest)
Traceback (most recent call last): File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line 55, in test_all_tables_use_innodb table), 'InnoDB') AssertionError: u'MyISAM' != 'InnoDB'
I've not created any new tables there (only added columns to existing tables). So how can this happen?
Sorry, not sure what you mean by that. Beaker uses the InnoDB storage engine, so MariaDB/MySQL should be configured appropriately [1].
Hmm, I configured it only following the developer part of patch testing [1]. There is only a snip about creating extra databases (which I think are created by the tests anyway). These two parts describing DB config are probably a good candidates for merging :).
https://beaker-project.org/dev/guide/writing-a-patch.html#testing-your-patch
[1] https://beaker-project.org/dev/guide/getting-started.html
So the main question here is, are all the tests expected to pass on the develop branch? Or what are the usual results I can compare my patches to? Are there any other requirements besides the deps of integration tests RPM and https://beaker-project.org/dev/guide/writing-a-patch.html#testing-your-patch...
On RHEL 6.6/ CentOS, they should all pass. On Fedora 20/21, you may have a few failures. We have a script to setup everything needed for running the tests: https://git.beaker-project.org/cgit/beaker/tree/Misc/beaker_test_setup.sh?h=...
But, it needs an update. The part where it downloads the Selenium JAR is no longer relevant.
Thanks, now I have a better idea what to expect.
Thank you, Jaroslav.
PS: are the results saved somewhere or is stdout the only option? "--debug-log" looks more like the framework debugging than the actual tests' debug output...
I think stdout is the only option.
:(
-- Jaroslav.
FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest)
Traceback (most recent call last): File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line 55, in test_all_tables_use_innodb table), 'InnoDB') AssertionError: u'MyISAM' != 'InnoDB'
I've not created any new tables there (only added columns to existing tables). So how can this happen?
Sorry, not sure what you mean by that. Beaker uses the InnoDB storage engine, so MariaDB/MySQL should be configured appropriately [1].
Hmm, I configured it only following the developer part of patch testing [1]. There is only a snip about creating extra databases (which I think are created by the tests anyway). These two parts describing DB config are probably a good candidates for merging :).
https://beaker-project.org/dev/guide/writing-a-patch.html#testing-your-patch
Here's the snip:
====================================================================== FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line 56, in test_all_tables_use_innodb table), 'InnoDB') AssertionError: u'MyISAM' != 'InnoDB' -------------------- >> begin captured stdout << --------------------- JKDEBUG: examining table activity JKDEBUG: examining table alembic_version
It seems that the alembic_version is a table created only by the tests and probably in some other manner that requires the defaults to be set in the mysql config, instead of what we are used to from the model :).
This is also the only table I can see there created with different type than InnoDB.
Cheers, Jaroslav.
Excerpts from Jaroslav Kortus's message of 2015-03-03 03:57 +10:00:
Here's the snip:
====================================================================== FAIL: test_all_tables_use_innodb (bkr.inttest.server.test_model.SchemaSanityTest)
Traceback (most recent call last): File "/root/beaker-private/IntegrationTests/src/bkr/inttest/server/test_model.py", line 56, in test_all_tables_use_innodb table), 'InnoDB') AssertionError: u'MyISAM' != 'InnoDB' -------------------- >> begin captured stdout << --------------------- JKDEBUG: examining table activity JKDEBUG: examining table alembic_version
It seems that the alembic_version is a table created only by the tests and probably in some other manner that requires the defaults to be set in the mysql config, instead of what we are used to from the model :).
Indeed! You have found a bug :-) Filed: https://bugzilla.redhat.com/show_bug.cgi?id=1197917
Workaround is to add the default-storage-engine=innodb setting to my.cnf.
beaker-devel@lists.fedorahosted.org