running conductor tests upstream

Matt Wagner matt.wagner at redhat.com
Mon Jun 18 22:24:45 UTC 2012


On Mon, Jun 18, 2012 at 12:56:48PM -0700, Richard Su wrote:
> I have been working on getting automated testing going for upstream.
> Mike asked me to share some of the results primarily to highlight the
> tests that are broken.

This is awesome. Thanks for doing it!

> 
> The tests were executed in a cloudbees hosted build environment
> (final environment is yet to be determined, we're considering other
> options that may be less costly). 

I honestly haven't tried it, so it may well not be suitable, but I
wonder if OpenShift's Jenkins support would help us here?
https://openshift.redhat.com/community/blogs/jenkins-and-more-new-openshift-release-nov-15-2011

> For upstream, Bundler was turned on
> and rpms were not used. aeolus-configure was also not executed, so
> services like deltacloud-core were off. sqlite was used in place of
> postgres.

Interesting -- a pure Ruby Conductor setup?

> I did a little bit a digging locally and 7 of these failures can be
> attributed to turning Bundler on.
> 
> Failing Scenarios:
> cucumber features/deployables.feature:43 # Scenario: Launch a deployment
> cucumber features/deployment.feature:169 # Scenario: Delete a deployment
> cucumber features/deployment.feature:176 # Scenario: Delete multiple
> deployments
> cucumber features/deployment.feature:189 # Scenario: Delete a
> deployment with running instances
> cucumber features/instance.feature:66 # Scenario: Stop instance
> cucumber features/instance.feature:85 # Scenario: Stop multiple instances
> cucumber features/provider.feature:74 # Scenario: Disable a provider

Only 7 failures actually seems surprisingly good to me here.

> 10 of these can be attributed to having deltacloud-core off, with all
> other services running and bundler off.
> 
> Failing Scenarios:
> cucumber features/deployment.feature:77 # Scenario: Stop deployments
> cucumber features/deployment.feature:97 # Scenario: Stop a deployment
> over XHR
> cucumber features/deployment.feature:139 # Scenario: Stop a deployment
> cucumber features/deployment.feature:169 # Scenario: Delete a deployment
> cucumber features/deployment.feature:176 # Scenario: Delete multiple
> deployments
> cucumber features/deployment.feature:189 # Scenario: Delete a
> deployment with running instances
> cucumber features/instance.feature:66 # Scenario: Stop instance
> cucumber features/instance.feature:85 # Scenario: Stop multiple instances
> cucumber features/instance.feature:202 # Scenario: Stop an instance
> cucumber features/provider.feature:74 # Scenario: Disable a provider

Not that you're saying otherwise, but this should be considered a bug in
our tests. We shouldn't be hitting live Deltacloud from unit tests.

> 11 of these were from both turning Bundler on and deltacloud-core
> off, with all other aeolus services on. So there are some overlap.
> 
> The remaining 6 failures are the result of other services being off
> or not executing aeolus-configure.

Like with Deltacloud, I don't think we should actually require any other
services to be running.

I wonder what to do about aeolus-configure. I know it does some database
setup, but that shouldn't matter for the test database. I wonder what
else it's doing here that tests depend on.

> For rspec, I got 187 failures. Most them are of this nature:
> 
> 187) Provider (unavailable provider) should mark provider as
> available if it becomes accessible
>      Failure/Error: @provider =
> FactoryGirl.create(:unavailable_mock_provider)
>      ActiveRecord::RecordInvalid:
>        Validation failed: Provider URL Must be a valid provider uri
>      # ./spec/models/provider_spec.rb:157

I think the majority of these failures are related to Deltacloud
dependencies as well. If I disable deltacloud-core and run "rake spec",
I get 180 failures locally. So stubbing/mocking/recording our Deltacloud
interactions should get us very far here.

Thanks for this -- I'm excited to see us moving in this direction!

-- Matt



More information about the aeolus-devel mailing list