Tim Flink tflink@redhat.com writes:
For RHEL cloud image checks we have special tool called 'valid': https://github.com/RedHatQE/valid , it can be reused for Fedora as well. The questions are: how do you see the integration and how can someone (me?) work on this (non-existent atm) part of Taskotron.
For now, I think that the best place to start would be to figure out what's missing from taskotron in order for the integration to happen.
- would the cloud image startup be done by valid or in taskotron?
- where would the cloud images be run?
- what other image work would be done in the task?
Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml file with all tests, their steps and results + logs). I think we can easily kick off such testing tasks from taskotron. The only missing piece in the picture is an 'image uploader': after image was built in koji someone needs to upload it as an AMI to EC2 (afaict Dennis does that manually now).
2mattdm: Matt, I remember you telling me that someone is working on such uploader. Can you please shed some light on what's the status now and what can be done in before f21?
I can see two possible approaches:
- 'Easy'. We create special type of task (sorry if I'm not that
precise with terminology) which is being executed on static slave node which runs valid (in black-box mode) on newly uploaded image, grabs the result and returns it back.
I suspect that this will be the best way to get started. The ephemeral client work you mention below is going to be a lot of work and there are a bunch of unknowns (use openstack or not, how isolated should the clients be etc.). If we want to have a shot at having some of the automated cloud tests done for f21, I think that starting with the easiest route would likely be best
Ok, I think that having something ready in f21 timeframe is really doable in case releng can automate AMIs uploading process. I'll take a look at the runner in libtaskotron and existing tasks examples.
Apologies for the delayed response. Looking forward to working with you on cloud image testing.
No prob)
On Thu, Apr 17, 2014 at 02:25:03PM +0200, Vitaly Kuznetsov wrote:
2mattdm: Matt, I remember you telling me that someone is working on such uploader. Can you please shed some light on what's the status now and what can be done in before f21?
Yeah. Still manual process. It feels like a very long time ago that we were talking about this in Brno! But David Gay (oddshocks) has been hired as a summer intern and will be working on this. Some related tickets:
https://fedorahosted.org/cloud/ticket/35 https://fedorahosted.org/cloud/ticket/38
Ok, I think that having something ready in f21 timeframe is really doable in case releng can automate AMIs uploading process. I'll take a look at the runner in libtaskotron and existing tasks examples.
Awesome!
On Thu, 17 Apr 2014 14:25:03 +0200 Vitaly Kuznetsov vitty@redhat.com wrote:
Tim Flink tflink@redhat.com writes:
For RHEL cloud image checks we have special tool called 'valid': https://github.com/RedHatQE/valid , it can be reused for Fedora as well. The questions are: how do you see the integration and how can someone (me?) work on this (non-existent atm) part of Taskotron.
For now, I think that the best place to start would be to figure out what's missing from taskotron in order for the integration to happen.
- would the cloud image startup be done by valid or in taskotron?
- where would the cloud images be run?
- what other image work would be done in the task?
Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml file with all tests, their steps and results + logs). I think we can easily kick off such testing tasks from taskotron. The only missing piece in the picture is an 'image uploader': after image was built in koji someone needs to upload it as an AMI to EC2 (afaict Dennis does that manually now).
The concern I have with this is around EC2 credentials and accounts. If I'm understanding you correctly, we would need to have those credentials stored on the clients and we'd be using EC2 time for testing Fedora.
Does someone have budget or account time that could be used for this in an account that we could use? I wonder if trying to use the fedora openstack setup be a better route (assuming that valid can work with openstack).
Another possible snag is that I want to start locking down network access on most if not all of the test clients so that it's less possible for user-submitted tasks to go awry and do things they shouldn't. This hasn't been done yet, though and it's something that we can discuss going forward.
2mattdm: Matt, I remember you telling me that someone is working on such uploader. Can you please shed some light on what's the status now and what can be done in before f21?
I can see two possible approaches:
- 'Easy'. We create special type of task (sorry if I'm not that
precise with terminology) which is being executed on static slave node which runs valid (in black-box mode) on newly uploaded image, grabs the result and returns it back.
I suspect that this will be the best way to get started. The ephemeral client work you mention below is going to be a lot of work and there are a bunch of unknowns (use openstack or not, how isolated should the clients be etc.). If we want to have a shot at having some of the automated cloud tests done for f21, I think that starting with the easiest route would likely be best
Ok, I think that having something ready in f21 timeframe is really doable in case releng can automate AMIs uploading process. I'll take a look at the runner in libtaskotron and existing tasks examples.
Cool, let us know if you have any questions or if anything isn't clear.
Tim
On Thu, Apr 17, 2014 at 09:11:10AM -0600, Tim Flink wrote:
Does someone have budget or account time that could be used for this in an account that we could use? I wonder if trying to use the fedora openstack setup be a better route (assuming that valid can work with openstack).
I think we can use the Fedora community cloud EC2 account for this, but I will double check. If that's not possible, I'll beg for the required budget.
We _should_ test on both OpenStack and EC2.
Tim Flink tflink@redhat.com writes:
On Thu, 17 Apr 2014 14:25:03 +0200 Vitaly Kuznetsov vitty@redhat.com wrote:
Tim Flink tflink@redhat.com writes:
For RHEL cloud image checks we have special tool called 'valid': https://github.com/RedHatQE/valid , it can be reused for Fedora as well. The questions are: how do you see the integration and how can someone (me?) work on this (non-existent atm) part of Taskotron.
For now, I think that the best place to start would be to figure out what's missing from taskotron in order for the integration to happen.
- would the cloud image startup be done by valid or in taskotron?
- where would the cloud images be run?
- what other image work would be done in the task?
Atm 'valid' gets EC2 AMI id as an input and produces test result (yaml file with all tests, their steps and results + logs). I think we can easily kick off such testing tasks from taskotron. The only missing piece in the picture is an 'image uploader': after image was built in koji someone needs to upload it as an AMI to EC2 (afaict Dennis does that manually now).
The concern I have with this is around EC2 credentials and accounts. If I'm understanding you correctly, we would need to have those credentials stored on the clients and we'd be using EC2 time for testing Fedora.
Does someone have budget or account time that could be used for this in an account that we could use? I wonder if trying to use the fedora openstack setup be a better route (assuming that valid can work with openstack).
It can't right now but switching from boto to something like libcloud is already in my backlog. In case we do have such requirement (and Matt's letter says we do) I can increase the priority for the task.
Another possible snag is that I want to start locking down network access on most if not all of the test clients so that it's less possible for user-submitted tasks to go awry and do things they shouldn't. This hasn't been done yet, though and it's something that we can discuss going forward.
For valid we'll require two things: 1) Access to Cloud's (AWS, Openstack, ...) endpoint 2) SSH to running VM
Can we have special dedicated test client for valid? That would make sense from securitty pov as we need to store cloud access credentials there.
2mattdm: Matt, I remember you telling me that someone is working on such uploader. Can you please shed some light on what's the status now and what can be done in before f21?
I can see two possible approaches:
- 'Easy'. We create special type of task (sorry if I'm not that
precise with terminology) which is being executed on static slave node which runs valid (in black-box mode) on newly uploaded image, grabs the result and returns it back.
I suspect that this will be the best way to get started. The ephemeral client work you mention below is going to be a lot of work and there are a bunch of unknowns (use openstack or not, how isolated should the clients be etc.). If we want to have a shot at having some of the automated cloud tests done for f21, I think that starting with the easiest route would likely be best
Ok, I think that having something ready in f21 timeframe is really doable in case releng can automate AMIs uploading process. I'll take a look at the runner in libtaskotron and existing tasks examples.
Cool, let us know if you have any questions or if anything isn't clear.
Tim
On Thu, 17 Apr 2014 17:35:13 +0200 Vitaly Kuznetsov vkuznets@redhat.com wrote:
<snip>
Another possible snag is that I want to start locking down network access on most if not all of the test clients so that it's less possible for user-submitted tasks to go awry and do things they shouldn't. This hasn't been done yet, though and it's something that we can discuss going forward.
For valid we'll require two things:
- Access to Cloud's (AWS, Openstack, ...) endpoint
- SSH to running VM
Interfacing with EC2 wasn't a use-case that I was thinking of for network isolation of the taskotron clients, so those plans may change somewhat. The clients aren't isolated yet, so this won't be a problem immediately.
Can we have special dedicated test client for valid? That would make sense from securitty pov as we need to store cloud access credentials there.
I suppose that we could but I'd really prefer to avoid that if at all possible. Having one "special" client isn't an issue but it does open the door to other task authors asking for the same thing and that will get unmanageable pretty quick.
That being said, I'm not sure how to go about managing credentials like that in a secure fashion. This'll require some more thought but suggestions are certainly welcome :)
Tim