[Fwd: Run through of aeolus cucumber tests]

Matt Wagner matt.wagner at redhat.com
Tue May 15 20:27:08 UTC 2012


Howdy,

Some replies inline below:

On Tue, May 15, 2012 at 03:07:05PM -0400, Dave Johnson wrote:
> So QE is looking into getting in on the existing cucumber action if it
> makes sense but first we have some questions that we would like to work
> through with dev.  Any assistance is appreciated.

Overall, I think I'm of two minds about this. I certainly think working
together on this is probably a good idea. I imagine it would save you
guys some time, and we could probably use some help with some tests.

At the same time, Cucumber tests are somewhat a developer's toolset.
It's important that developers write them as they go. (Although
admittedly, we don't always do a very good job of this, myself
included.) I hardly mean to suggest that you're not allowed to touch
them, just that I want to make sure that we don't fall into a trap of
thinking we don't have to write tests because QE will do it for us.

(Though our tests are really slow right now, so I'm not crazy about
adding lots more without good reason.)

More details below:

> 
> Thanks,
> -Dave 

> Date: Mon, 14 May 2012 18:25:17 -0400 (EDT)
> From: Ronelle Landy <rlandy at redhat.com>
> To: Jason Guiditta <jguiditt at redhat.com>
> Cc: wes hayutin <whayutin at redhat.com>, Dave Johnson <dajohnso at redhat.com>
> Subject: Run through of aeolus cucumber tests
> 
> Hi Jason,
> 
> Wes, Dave and I have been looking through the aeolus cucumber tests with an eye to including some QE tests within the framework dev has been using. We had some questions wrt how the tests are run what types of testing can be done effectively with cucumber. For example:
> 
>  - How is dev working with VCR? Can we rerecord with each test run so we are not working with any possibly stale features/objects/data?

We use VCR as a way of mocking requests, so we don't have to make live
API calls in our tests. Part of the reason we do this is so that you
*don't* run into stale data.

>  - Looks like the dev tests are using the mock provider. Is it possible to test all functionality available for mock or are there other limitations?

I'm not sure. Our goal isn't to necessarily test the driver fully --
that's Deltacloud's problem. We just want to test the bits where we
interact with it.

>  - Would it be suitable for QE to submit additions directly into the src directories?

My kind-of-rambling comments at the top of this email relate to this. I
think the answer is that it might be valuable for you to send patches,
with the caveat that our needs might not always align so you shouldn't
take it personally if we don't accept some of them. I think it really
depends on exactly what you're doing.

>  - Are there areas where you see the cucumber tests pass but the scenario could fail when a user actually interacted with the application?

Ideally, the answer is "No" here. However, I think our aims are slightly
different -- our tests are more focused on basic functionality and
proving that we didn't break anything with a new patch, versus
explicitly checking weird edge cases that a user could cause. (Though
this isn't to say that we shouldn't test those things where it makes
sense.)

>  - How is selenium used? 

I don't believe we explicitly use it at all, but it may be an underlying
dependency. I'm not the most qualified to comment on this one.

-- Matt



More information about the aeolus-devel mailing list