running conductor tests upstream

Michael Orazi morazi at redhat.com
Tue Jun 19 19:19:10 UTC 2012



----- Original Message -----
> On Mon, Jun 18, 2012 at 12:56:48PM -0700, Richard Su wrote:
> > I have been working on getting automated testing going for
> > upstream.
> > Mike asked me to share some of the results primarily to highlight
> > the
> > tests that are broken.
> 
> This is awesome. Thanks for doing it!
> 

Yea, +1000.  Lvoe that we will eventually get the community something world visible!

> > 
> > The tests were executed in a cloudbees hosted build environment
> > (final environment is yet to be determined, we're considering other
> > options that may be less costly).
> 
> I honestly haven't tried it, so it may well not be suitable, but I
> wonder if OpenShift's Jenkins support would help us here?
> https://openshift.redhat.com/community/blogs/jenkins-and-more-new-openshift-release-nov-15-2011
> 

That was on the short-list:
- cloudbees
- openshift/jenkins
- travis-ci

I think I originally misread the blog you mentioned above and thought it might not be as suitable for us, but re-reading I don't remember my original concern...

> > For upstream, Bundler was turned on
> > and rpms were not used. aeolus-configure was also not executed, so
> > services like deltacloud-core were off. sqlite was used in place of
> > postgres.
> 
> Interesting -- a pure Ruby Conductor setup?
> 
> > I did a little bit a digging locally and 7 of these failures can be
> > attributed to turning Bundler on.
> > 
> > Failing Scenarios:
> > cucumber features/deployables.feature:43 # Scenario: Launch a
> > deployment
> > cucumber features/deployment.feature:169 # Scenario: Delete a
> > deployment
> > cucumber features/deployment.feature:176 # Scenario: Delete
> > multiple
> > deployments
> > cucumber features/deployment.feature:189 # Scenario: Delete a
> > deployment with running instances
> > cucumber features/instance.feature:66 # Scenario: Stop instance
> > cucumber features/instance.feature:85 # Scenario: Stop multiple
> > instances
> > cucumber features/provider.feature:74 # Scenario: Disable a
> > provider
> 
> Only 7 failures actually seems surprisingly good to me here.
> 
> > 10 of these can be attributed to having deltacloud-core off, with
> > all
> > other services running and bundler off.
> > 
> > Failing Scenarios:
> > cucumber features/deployment.feature:77 # Scenario: Stop
> > deployments
> > cucumber features/deployment.feature:97 # Scenario: Stop a
> > deployment
> > over XHR
> > cucumber features/deployment.feature:139 # Scenario: Stop a
> > deployment
> > cucumber features/deployment.feature:169 # Scenario: Delete a
> > deployment
> > cucumber features/deployment.feature:176 # Scenario: Delete
> > multiple
> > deployments
> > cucumber features/deployment.feature:189 # Scenario: Delete a
> > deployment with running instances
> > cucumber features/instance.feature:66 # Scenario: Stop instance
> > cucumber features/instance.feature:85 # Scenario: Stop multiple
> > instances
> > cucumber features/instance.feature:202 # Scenario: Stop an instance
> > cucumber features/provider.feature:74 # Scenario: Disable a
> > provider
> 
> Not that you're saying otherwise, but this should be considered a bug
> in
> our tests. We shouldn't be hitting live Deltacloud from unit tests.
> 

Yea, I asked Richard to report it here simply for discussion sake.  My hope is that these wind up as tracked items in the upstream tracker and they start getting addressed in short order.

> > 11 of these were from both turning Bundler on and deltacloud-core
> > off, with all other aeolus services on. So there are some overlap.
> > 
> > The remaining 6 failures are the result of other services being off
> > or not executing aeolus-configure.
> 
> Like with Deltacloud, I don't think we should actually require any
> other
> services to be running.
> 

Agreed.

> I wonder what to do about aeolus-configure. I know it does some
> database
> setup, but that shouldn't matter for the test database. I wonder what
> else it's doing here that tests depend on.
> 

re: aeolus-configure, we can typically replicate what needs replicating via a few additional rake calls inside the jenkins job.  

> > For rspec, I got 187 failures. Most them are of this nature:
> > 
> > 187) Provider (unavailable provider) should mark provider as
> > available if it becomes accessible
> >      Failure/Error: @provider =
> > FactoryGirl.create(:unavailable_mock_provider)
> >      ActiveRecord::RecordInvalid:
> >        Validation failed: Provider URL Must be a valid provider uri
> >      # ./spec/models/provider_spec.rb:157
> 
> I think the majority of these failures are related to Deltacloud
> dependencies as well. If I disable deltacloud-core and run "rake
> spec",
> I get 180 failures locally. So stubbing/mocking/recording our
> Deltacloud
> interactions should get us very far here.
> 
> Thanks for this -- I'm excited to see us moving in this direction!
> 
> -- Matt
> 

Exactly -- I was doing some poking around locally and have similar numbers when I turn off imagefactory/iwhd/deltacloud-core.  My guess is that a few mocks will get us a long way.

I'd say that my personal opinion is that mocks or stubs seem like the more correct thing to do for rspec, which I equate with pure unit testing.  For cucubmer tests, which I think of as typically slightly closer to integration testing recording seems like the right thing to do.

m



More information about the aeolus-devel mailing list