On 01/11/11 11:07 -0400, Matt Wagner wrote:
On Tue, Nov 01, 2011 at 10:36:29AM -0400, Scott Seago wrote:
> On 11/01/2011 08:48 AM, Chris Alfonso wrote:
> >It's also worth mentioning, the deployment tests take a really long
> >time to execute, so by marking them with a tag that is by default
> >excluded, you can get through a test run much more quickly. We may
> >want to introduce a new tag with @takesforever or something, that
> >way you could run sanity check tests more often, then run the
> >@takesforever tests when you really want to test everything.
> Hmm,
>
> I know we need to mock core so that it doesn't need to be running
> (VCR perhaps?), but marking them all as @wip now means that, until we
> remove this dep, people will stop testing them, and errors/failures
> will start to creep in. Is that the best thing to do here? Or is
> someone actively working on removing this dep right now -- which is
> at least vaguely implied by the "in progress" part of @wip.
I think the work in progress part is getting vcr setup for this. right now the
tests faile if the service isn't running. I don't think @wip means keep the tag
there and ignore the test, it means keep working on it. I think we need a story to
represent the work to get the VCR setup done. I didn't want to comment out the test
because I think we should be able to rake cucumber cucumber:wip if we really want to run
the tests that require deltacloud-core running.
I like the idea of having an 'express' set of tests that can
run in a
couple minutes, and then a 'thorough' set of tests that exercise full
functionality.
Yeah, the @wip is part of cucumber and part of the rake tasks
already. Nothing stopping us from having our own tags too.
But I do rather share Scott's concern that people, whether by omission
or just being unaware, will forget to run the more involved tests at
all if it's not standard.
Do you think it would make sense to create a couple rake tasks, where
the default is to run everything? e.g.,
rake test # runs everything
rake test:quick # abbreviated set
Yeah, I agree with both of you on this.