running conductor tests upstream
Matt Wagner
matt.wagner at redhat.com
Mon Jun 18 22:24:45 UTC 2012
On Mon, Jun 18, 2012 at 12:56:48PM -0700, Richard Su wrote:
> I have been working on getting automated testing going for upstream.
> Mike asked me to share some of the results primarily to highlight the
> tests that are broken.
This is awesome. Thanks for doing it!
>
> The tests were executed in a cloudbees hosted build environment
> (final environment is yet to be determined, we're considering other
> options that may be less costly).
I honestly haven't tried it, so it may well not be suitable, but I
wonder if OpenShift's Jenkins support would help us here?
https://openshift.redhat.com/community/blogs/jenkins-and-more-new-openshift-release-nov-15-2011
> For upstream, Bundler was turned on
> and rpms were not used. aeolus-configure was also not executed, so
> services like deltacloud-core were off. sqlite was used in place of
> postgres.
Interesting -- a pure Ruby Conductor setup?
> I did a little bit a digging locally and 7 of these failures can be
> attributed to turning Bundler on.
>
> Failing Scenarios:
> cucumber features/deployables.feature:43 # Scenario: Launch a deployment
> cucumber features/deployment.feature:169 # Scenario: Delete a deployment
> cucumber features/deployment.feature:176 # Scenario: Delete multiple
> deployments
> cucumber features/deployment.feature:189 # Scenario: Delete a
> deployment with running instances
> cucumber features/instance.feature:66 # Scenario: Stop instance
> cucumber features/instance.feature:85 # Scenario: Stop multiple instances
> cucumber features/provider.feature:74 # Scenario: Disable a provider
Only 7 failures actually seems surprisingly good to me here.
> 10 of these can be attributed to having deltacloud-core off, with all
> other services running and bundler off.
>
> Failing Scenarios:
> cucumber features/deployment.feature:77 # Scenario: Stop deployments
> cucumber features/deployment.feature:97 # Scenario: Stop a deployment
> over XHR
> cucumber features/deployment.feature:139 # Scenario: Stop a deployment
> cucumber features/deployment.feature:169 # Scenario: Delete a deployment
> cucumber features/deployment.feature:176 # Scenario: Delete multiple
> deployments
> cucumber features/deployment.feature:189 # Scenario: Delete a
> deployment with running instances
> cucumber features/instance.feature:66 # Scenario: Stop instance
> cucumber features/instance.feature:85 # Scenario: Stop multiple instances
> cucumber features/instance.feature:202 # Scenario: Stop an instance
> cucumber features/provider.feature:74 # Scenario: Disable a provider
Not that you're saying otherwise, but this should be considered a bug in
our tests. We shouldn't be hitting live Deltacloud from unit tests.
> 11 of these were from both turning Bundler on and deltacloud-core
> off, with all other aeolus services on. So there are some overlap.
>
> The remaining 6 failures are the result of other services being off
> or not executing aeolus-configure.
Like with Deltacloud, I don't think we should actually require any other
services to be running.
I wonder what to do about aeolus-configure. I know it does some database
setup, but that shouldn't matter for the test database. I wonder what
else it's doing here that tests depend on.
> For rspec, I got 187 failures. Most them are of this nature:
>
> 187) Provider (unavailable provider) should mark provider as
> available if it becomes accessible
> Failure/Error: @provider =
> FactoryGirl.create(:unavailable_mock_provider)
> ActiveRecord::RecordInvalid:
> Validation failed: Provider URL Must be a valid provider uri
> # ./spec/models/provider_spec.rb:157
I think the majority of these failures are related to Deltacloud
dependencies as well. If I disable deltacloud-core and run "rake spec",
I get 180 failures locally. So stubbing/mocking/recording our Deltacloud
interactions should get us very far here.
Thanks for this -- I'm excited to see us moving in this direction!
-- Matt
More information about the aeolus-devel
mailing list