Say I have a resource with a Enable/Disable operation that also has a Status trait, e.g. a
Deployment/application service under AS7.
Here are some scenarios that I've tried out (apologies if I've mis-remembered
something here, I've not run the through the whole battery of scenarios after my
initial tests, but I think the following are accurate)
1a) Shutting down the deployment (trait checking faster than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but
change its Status trait to be collected every 30seconds.
Deployment starts off green and status = OK
Execute the disable operation. It completes successfully very quickly.
Wait 1min
The status trait now says DOWN, but the availability still says green, but the app is
actually not available.
Wait some more time, possibly upto 10mins.
The availability will now show red.
1b) Shutting down the deployment (trait checking slower than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but
change its Status trait to be collected every 20minutes
Deployment starts off green and status = OK
Execute the disable operation. It completes successfully very quickly.
Wait 1min
The status trait says OK and the availability still says green, but the app is actually
not available.
Wait some more time, possibly upto 10mins.
The availability will now show red, but (assuming the trait collection still hasn't
fired) the status trait still says OK.
The status trait will now remain OK forever or until we start collecting availability
again.
2a) Starting up the deployment (trait checking faster than availability)
---------------------------------
I leave the deployment with its default availability collection period of 10mins but
change its Status trait to be collected every 30seconds.
Deployment starts off red and status = DOWN
Execute the enable operation. It completes successfully very quickly.
Wait 1min
The status trait says DOWN and the availability still says red, but the app is actually
available.
Wait some more time, possibly upto 10mins.
The availability will now show green. Shortly after that the status will go to OK
2b) Starting up the deployment (trait checking slower than availability)
---------------------------------------
I leave the deployment with its default availability collection period of 10mins but
change its Status trait to be collected every 20minutes
Deployment starts off red and status = DOWN
Execute the enable operation. It completes successfully very quickly.
Wait 1min
The status trait says DOWN and the availability still says red, but the app is actually
available.
Wait some more time, possibly upto 10mins.
The availability will now show green, but (assuming the trait collection still hasn't
fired) the status trait still says DOWN.
Wait some more time, possibly upto another 10mins.
The status trait will eventually flip to OK
Summary and thoughts:
i) IMHO its not very intuitive that having various different schedules for availability
and trait collection can result in radically different results. I think the general idea
around metric/avail/traits should be that if the system ends up in a steady state that the
metric/avail/trait should eventually get into that same steady state. The frequency of
collection should just determine how quickly its get there.
So looking at the results above I see the following:
2b) is intuitive, slow trait schedule => get to steady state slowly
1a) is intuitive, fast trait schedule => get to steady state quickly
2a) is not intuitive, fast trait schedule doesn't update the trait quickly
1b) looks broken, trait may never get into correct state
I'm not sure what could be done about the last two. 1b) could possibly be mitigated by
letting metrics and traits be collected one final time after the avail has gone red. 2a)
could perhaps be addressed to a limited extent by updating what happens when an operation
executes, as discussed below.
ii) For operations launched from the RHQ CLI/UI there should be a way (if there isnt
already) to trigger an availability/trait/metric collection after the operation has
completed. [If we already have this, then the AS7 plugin needs to be updated to support
it]
That way users will see the very latest state as soon as possible after the operation is
done. Its obviously fine for out of band events, e.g. someone manually deleting the
application from the filesystem, to get picked up by the scheduled polling which the
plugin does.
What does everyone think? BTW I'm not arguing about why we've ended up where we
have, e.g. not collecting metrics/traits for down resources makes perfect sense from a
performance perspective. I just want us to think about how we can improve going forward.
Cheers
Charles