RFC: Dbomatic testability and performance

Jan Provaznik jprovazn at redhat.com
Tue Apr 3 14:13:25 UTC 2012


On 04/03/2012 01:25 AM, Richard Su wrote:
> Hello,
>
> Here's a start of a feature outline. Contents of the wiki page is listed
> below.
> https://www.aeolusproject.org/redmine/projects/aeolus/wiki/Dbomatic_testability_and_performance
>
>
> What I am unsure of is which part of dbomatic performance needs
> addressing? From my limited experience, I have seen potential issues
> with RHEV and the speed in which it starts instances. But that may have
> been specific to the environment I was using.
>
> If you have experienced issues with dbomatic, this would be a great time
> to get them out in the open so that we may include them in this
> refactoring of dbomatic.
>
> Thanks.
>
> ----
>
> # Summary
>
> Dbomatic is currently responsible for updating instance statuses and ip
> addresses and updating realms. It also starts freshly created RHEV
> instances.
>
> We do not have any test coverage for dbomatic and it is written in a way
> that makes testing it difficult. We would like to refactor dbomatic to
> make it more readable and in a way that allows us to plug in tests.
>
> As part of the refactoring, we should also explore ways to improve its
> performance. For example, there is currently a 30 second time window
> allocated for each provider account where dbomatic communicates with
> deltacloud to perform status update, launch instances, etc. This 30
> second window may be too short especially if dbomatic is asked to launch
> a large number of RHEV instances.
>
> # Owner
>
> Richard Su (rwsu)
> and others who have interest
>
> # Current status
>
> Planning phase
>
> # Screencast Demo
>
> Run dbomatic tests.
> Demonstrate dbomatic can handle status updates and launches where they
> are a large number of instances. This can be through a unit test.
>
> # Implementation tasks
>
> This feature is being tracked at task #3094.
>
> Tests to add
>
> * State change reported by deltacloud is then reported in dbomatic
> * Verify extraction and population of public and private ip addresses
> * For RHEV, verify instance that is in STOPPED state and had previously
> not been in running state is then started
> * Realms refresh
> * When provider is unavailable, dbomatic stays up and doesn't fail and exit
>
> Performance related tasks
>
> * Dbomatic should not timeout if is asked to launch a large number of
> RHEV instances. Mock out a delay to simulate RHEV instance starts.
> Eliminate the 30 second deltacloud timeouts.
> * Profile RHEV instance launch performance from dbomatic, and optimize
> code where needed.
>
> # Open Questions
>
> * Which part of dbomatic performance are we worried about?


Sooner today I proposed in thread "RFC feature planning - robust 
instance launching" to replace dbomatic with a background job tool which 
supports both delayed and recurring jobs:
- delayed jobs would be used for launching instances
- recurring jobs would be used for all stuff done by dbomatic

The code in "check_one_account" method could be part of ProviderAccount 
model, so it can be covered by common rspec tests we have.
The rest of script is just about parsing params, and forking processes 
in an interval - we get all this logic for free when using a bg job tool.

Jan



More information about the aeolus-devel mailing list