Questions around traits/availability/operations

Charles Crouch ccrouch at redhat.com
Fri Jul 6 20:42:53 UTC 2012


Say I have a resource with a Enable/Disable operation that also has a Status trait, e.g. a Deployment/application service under AS7.
Here are some scenarios that I've tried out (apologies if I've mis-remembered something here, I've not run the through the whole battery of scenarios after my initial tests, but I think the following are accurate)


1a) Shutting down the deployment (trait checking faster than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but change its Status trait to be collected every 30seconds.
Deployment starts off green and status = OK
Execute the disable operation. It completes successfully very quickly.
Wait 1min
The status trait now says DOWN, but the availability still says green, but the app is actually not available.
Wait some more time, possibly upto 10mins.
The availability will now show red.


1b)  Shutting down the deployment (trait checking slower than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but change its Status trait to be collected every 20minutes
Deployment starts off green and status = OK
Execute the disable operation. It completes successfully very quickly.
Wait 1min
The status trait says OK and the availability still says green, but the app is actually not available.
Wait some more time, possibly upto 10mins.
The availability will now show red, but (assuming the trait collection still hasn't fired) the status trait still says OK.
The status trait will now remain OK forever or until we start collecting availability again.


2a) Starting up the deployment  (trait checking faster than availability)
---------------------------------
I leave the deployment with its default availability collection period of 10mins but change its Status trait to be collected every 30seconds.
Deployment starts off red and status = DOWN
Execute the enable operation. It completes successfully very quickly.
Wait 1min
The status trait says DOWN and the availability still says red, but the app is actually available.
Wait some more time, possibly upto 10mins.
The availability will now show green. Shortly after that the status will go to OK


2b)  Starting up the deployment (trait checking slower than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but change its Status trait to be collected every 20minutes
Deployment starts off red and status = DOWN
Execute the enable operation. It completes successfully very quickly.
Wait 1min
The status trait says DOWN and the availability still says red, but the app is actually available.
Wait some more time, possibly upto 10mins.
The availability will now show green, but (assuming the trait collection still hasn't fired) the status trait still says DOWN.
Wait some more time, possibly upto another 10mins.
The status trait will eventually flip to OK


Summary and thoughts:

i) IMHO its not very intuitive that having various different schedules for availability and trait collection can result in radically different results. I think the general idea around metric/avail/traits should be that if the system ends up in a steady state that the metric/avail/trait should eventually get into that same steady state. The frequency of collection should just determine how quickly its get there.
So looking at the results above I see the following:

2b) is intuitive, slow trait schedule => get to steady state slowly
1a) is intuitive, fast trait schedule => get to steady state quickly
2a) is not intuitive, fast trait schedule doesn't update the trait quickly
1b) looks broken, trait may never get into correct state

I'm not sure what could be done about the last two. 1b) could possibly be mitigated by letting metrics and traits be collected one final time after the avail has gone red. 2a) could perhaps be addressed to a limited extent by updating what happens when an operation executes, as discussed below.


ii) For operations launched from the RHQ CLI/UI there should be a way (if there isnt already) to trigger an availability/trait/metric collection after the operation has completed. [If we already have this, then the AS7 plugin needs to be updated to support it]
That way users will see the very latest state as soon as possible after the operation is done. Its obviously fine for out of band events, e.g. someone manually deleting the application from the filesystem, to get picked up by the scheduled polling which the plugin does.


What does everyone think? BTW I'm not arguing about why we've ended up where we have, e.g. not collecting metrics/traits for down resources makes perfect sense from a performance perspective. I just want us to think about how we can improve going forward.

Cheers
Charles




More information about the rhq-devel mailing list