Greetings,
Since I have an extra virt autotest-server setup, I decided to run through a few post-bodhi-update tests using the autoqa-0.3.5-pre1 package [1].
A few observations, some are probably known/expected, but I'll list them here for others.
1. I needed to create /usr/share/autoqa/post-bodhi-update/testlist for /usr/bin/autoqa to recognize the watcher. I just added rpmlint to test (see step#4 for results) 2. See attached output from running watch-bodhi-requests.py for the first time. Nothing horrible here. Perhaps reducing the error output produced by calling load_update_ids() when no cachefiles are present? 3. Attempting to schedule an autotest job using watch-bodhi-requests.py fails due to incorrect command-line arguments (see below). The attached patch resolves that problem. 4. Re-running the '/usr/bin/autoqa' using the hook patch from step#3 schedules an autotest job using the attached control file. Obviously, the job.run_test() call for rpmlint, doesn't use the same arguments that post-bodhi-update is creating. So 'rpmlint' might not be a good test. I'll try using the 'depcheck' test next.
Thanks, James
[1] https://fedorahosted.org/pipermail/autoqa-devel/2010-June/000687.html
On Tue, 2010-06-15 at 15:59 -0400, James Laska wrote:
Greetings,
Since I have an extra virt autotest-server setup, I decided to run through a few post-bodhi-update tests using the autoqa-0.3.5-pre1 package [1].
A few observations, some are probably known/expected, but I'll list them here for others.
1. I needed to create /usr/share/autoqa/post-bodhi-update/testlist for /usr/bin/autoqa to recognize the watcher. I just added rpmlint to test (see step#4 for results)
I was thinking - maybe we should have a really stupid "helloworld"-style test that just basically logs its inputs and reports success. This would allow us to test new hooks without needing to find an appropriate test.
2. See attached output from running watch-bodhi-requests.py for the first time. Nothing horrible here. Perhaps reducing the error output produced by calling load_update_ids() when no cachefiles are present?
Yeah, I intended that we would remove/quiet that code once we were satisfied that the watcher wasn't doing anything stupid.
3. Attempting to schedule an autotest job using watch-bodhi-requests.py fails due to incorrect command-line arguments (see below). The attached patch resolves that problem.
Ah doh! I forgot to change that. Patch looks fine; please apply.
4. Re-running the '/usr/bin/autoqa' using the hook patch from step#3 schedules an autotest job using the attached control file. Obviously, the job.run_test() call for rpmlint, doesn't use the same arguments that post-bodhi-update is creating. So 'rpmlint' might not be a good test. I'll try using the 'depcheck' test next.
So maybe let's write that "helloworld" test, and try that out instead of trying to get depcheck working first?
All it would need to do is something vaguely like:
def run_once(self, *args, **kwargs): msg = "args = %s\nkwargs = %s" % (args, kwargs) self.results = msg print msg # maybe email the message here just to be sure email is working # etc.
...except writing the control file might be a bit tricky since we don't necessarily know the key names. hrm.
So Maybe autoqa ought to be writing all the hook arguments into a dict instead of individual args, so we can pass along all the autoqa arguments without knowing their names.
This is a valuable exercise, though - we're starting to have tests that could apply to multiple hooks (rpmlint for example) and we're going to need ways to handle that.
For example we might also need to standardize on argument names - so (e.g.) --target-tag will always mean "this is the tag that this package is trying to enter" regardless of whether we're doing post-koji-build or post-bodhi-update or whatever.
In the meantime, though, we can just write this 'echo_inputs' test specifically for post-bodhi-update, and worry about how to make it work for multiple hooks later.
So. Anyone want to take a crack at writing this test?
-w
On Wed, 2010-06-23 at 19:37 -0400, Will Woods wrote:
On Tue, 2010-06-15 at 15:59 -0400, James Laska wrote:
Greetings,
Since I have an extra virt autotest-server setup, I decided to run through a few post-bodhi-update tests using the autoqa-0.3.5-pre1 package [1].
A few observations, some are probably known/expected, but I'll list them here for others.
1. I needed to create /usr/share/autoqa/post-bodhi-update/testlist for /usr/bin/autoqa to recognize the watcher. I just added rpmlint to test (see step#4 for results)
I was thinking - maybe we should have a really stupid "helloworld"-style test that just basically logs its inputs and reports success. This would allow us to test new hooks without needing to find an appropriate test.
Not a bad idea!
2. See attached output from running watch-bodhi-requests.py for the first time. Nothing horrible here. Perhaps reducing the error output produced by calling load_update_ids() when no cachefiles are present?
Yeah, I intended that we would remove/quiet that code once we were satisfied that the watcher wasn't doing anything stupid.
3. Attempting to schedule an autotest job using watch-bodhi-requests.py fails due to incorrect command-line arguments (see below). The attached patch resolves that problem.
Ah doh! I forgot to change that. Patch looks fine; please apply.
Thanks. I've applied and pushed the patch into your wwoods branch. Hope I did that right.
http://git.fedorahosted.org/git/?p=autoqa.git;a=commit;h=f49163a26db7ee52e2a...
4. Re-running the '/usr/bin/autoqa' using the hook patch from step#3 schedules an autotest job using the attached control file. Obviously, the job.run_test() call for rpmlint, doesn't use the same arguments that post-bodhi-update is creating. So 'rpmlint' might not be a good test. I'll try using the 'depcheck' test next.
So maybe let's write that "helloworld" test, and try that out instead of trying to get depcheck working first?
All it would need to do is something vaguely like:
def run_once(self, *args, **kwargs): msg = "args = %s\nkwargs = %s" % (args, kwargs) self.results = msg print msg # maybe email the message here just to be sure email is working # etc.
...except writing the control file might be a bit tricky since we don't necessarily know the key names. hrm.
So Maybe autoqa ought to be writing all the hook arguments into a dict instead of individual args, so we can pass along all the autoqa arguments without knowing their names.
This is a valuable exercise, though - we're starting to have tests that could apply to multiple hooks (rpmlint for example) and we're going to need ways to handle that.
For example we might also need to standardize on argument names - so (e.g.) --target-tag will always mean "this is the tag that this package is trying to enter" regardless of whether we're doing post-koji-build or post-bodhi-update or whatever.
In the meantime, though, we can just write this 'echo_inputs' test specifically for post-bodhi-update, and worry about how to make it work for multiple hooks later.
I really like this idea.
So. Anyone want to take a crack at writing this test?
I'd love to, will see about workload. Looks like Kamil has the ticket filed so anyone can feel free to jump on this time permitting.
Thanks, James
autoqa-devel@lists.fedorahosted.org