I didn't read this whole thing yet but... as of RHQ today:
"its default availability collection period of 10mins"
is incorrect. Our default avail collection period is 1 minute now.
----- Original Message -----
Say I have a resource with a Enable/Disable operation that also has
a
Status trait, e.g. a Deployment/application service under AS7.
Here are some scenarios that I've tried out (apologies if I've
mis-remembered something here, I've not run the through the whole
battery of scenarios after my initial tests, but I think the
following are accurate)
1a) Shutting down the deployment (trait checking faster than
availability)
---------------------------------------
I leave the deployment with its default availability collection
period of 10mins but change its Status trait to be collected every
30seconds.
Deployment starts off green and status = OK
Execute the disable operation. It completes successfully very
quickly.
Wait 1min
The status trait now says DOWN, but the availability still says
green, but the app is actually not available.
Wait some more time, possibly upto 10mins.
The availability will now show red.
1b) Shutting down the deployment (trait checking slower than
availability)
---------------------------------------
I leave the deployment with its default availability collection
period of 10mins but change its Status trait to be collected every
20minutes
Deployment starts off green and status = OK
Execute the disable operation. It completes successfully very
quickly.
Wait 1min
The status trait says OK and the availability still says green, but
the app is actually not available.
Wait some more time, possibly upto 10mins.
The availability will now show red, but (assuming the trait
collection still hasn't fired) the status trait still says OK.
The status trait will now remain OK forever or until we start
collecting availability again.
2a) Starting up the deployment (trait checking faster than
availability)
---------------------------------
I leave the deployment with its default availability collection
period of 10mins but change its Status trait to be collected every
30seconds.
Deployment starts off red and status = DOWN
Execute the enable operation. It completes successfully very quickly.
Wait 1min
The status trait says DOWN and the availability still says red, but
the app is actually available.
Wait some more time, possibly upto 10mins.
The availability will now show green. Shortly after that the status
will go to OK
2b) Starting up the deployment (trait checking slower than
availability)
---------------------------------------
I leave the deployment with its default availability collection
period of 10mins but change its Status trait to be collected every
20minutes
Deployment starts off red and status = DOWN
Execute the enable operation. It completes successfully very quickly.
Wait 1min
The status trait says DOWN and the availability still says red, but
the app is actually available.
Wait some more time, possibly upto 10mins.
The availability will now show green, but (assuming the trait
collection still hasn't fired) the status trait still says DOWN.
Wait some more time, possibly upto another 10mins.
The status trait will eventually flip to OK
Summary and thoughts:
i) IMHO its not very intuitive that having various different
schedules for availability and trait collection can result in
radically different results. I think the general idea around
metric/avail/traits should be that if the system ends up in a steady
state that the metric/avail/trait should eventually get into that
same steady state. The frequency of collection should just determine
how quickly its get there.
So looking at the results above I see the following:
2b) is intuitive, slow trait schedule => get to steady state slowly
1a) is intuitive, fast trait schedule => get to steady state quickly
2a) is not intuitive, fast trait schedule doesn't update the trait
quickly
1b) looks broken, trait may never get into correct state
I'm not sure what could be done about the last two. 1b) could
possibly be mitigated by letting metrics and traits be collected one
final time after the avail has gone red. 2a) could perhaps be
addressed to a limited extent by updating what happens when an
operation executes, as discussed below.
ii) For operations launched from the RHQ CLI/UI there should be a way
(if there isnt already) to trigger an availability/trait/metric
collection after the operation has completed. [If we already have
this, then the AS7 plugin needs to be updated to support it]
That way users will see the very latest state as soon as possible
after the operation is done. Its obviously fine for out of band
events, e.g. someone manually deleting the application from the
filesystem, to get picked up by the scheduled polling which the
plugin does.
What does everyone think? BTW I'm not arguing about why we've ended
up where we have, e.g. not collecting metrics/traits for down
resources makes perfect sense from a performance perspective. I just
want us to think about how we can improve going forward.
Cheers
Charles
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/rhq-devel