Questions around traits/availability/operations

John Mazzitelli mazz at redhat.com
Fri Jul 6 21:16:02 UTC 2012


> not collecting metrics/traits for down
> resources makes perfect sense from a performance perspective.

Why assume that we could ever collect a metric or trait value from a resource that is down? In general, if a resource is down, we can't query it to know anything about it. Now, in your particular case, the trait (status) isn't really a value that is retrieved from the resource, so you could technically ask the plugin for it and you could get a valid value back, but that is a very specific implementation of that trait - why think in the general case that this is always true? If a resource is down, in general, assume you can know nothing about it since it won't be up to respond to the plugin's queries about its state.

What we could do is add a new piece of metadata to <metric> - that being something like "collectEvenIfResourceIsDown=true" - if true, I guess we might be able to change the MeasurementManager to make the call to collect that metric even if the resource is down.

----- Original Message -----
> I didn't read this whole thing yet but... as of RHQ today:
> 
> "its default availability collection period of 10mins"
> 
> is incorrect. Our default avail collection period is 1 minute now.
> 
> ----- Original Message -----
> > Say I have a resource with a Enable/Disable operation that also has
> > a
> > Status trait, e.g. a Deployment/application service under AS7.
> > Here are some scenarios that I've tried out (apologies if I've
> > mis-remembered something here, I've not run the through the whole
> > battery of scenarios after my initial tests, but I think the
> > following are accurate)
> > 
> > 
> > 1a) Shutting down the deployment (trait checking faster than
> > availability)
> > ---------------------------------------
> > I leave the deployment with its default availability collection
> > period of 10mins but change its Status trait to be collected every
> > 30seconds.
> > Deployment starts off green and status = OK
> > Execute the disable operation. It completes successfully very
> > quickly.
> > Wait 1min
> > The status trait now says DOWN, but the availability still says
> > green, but the app is actually not available.
> > Wait some more time, possibly upto 10mins.
> > The availability will now show red.
> > 
> > 
> > 1b)  Shutting down the deployment (trait checking slower than
> > availability)
> > ---------------------------------------
> > I leave the deployment with its default availability collection
> > period of 10mins but change its Status trait to be collected every
> > 20minutes
> > Deployment starts off green and status = OK
> > Execute the disable operation. It completes successfully very
> > quickly.
> > Wait 1min
> > The status trait says OK and the availability still says green, but
> > the app is actually not available.
> > Wait some more time, possibly upto 10mins.
> > The availability will now show red, but (assuming the trait
> > collection still hasn't fired) the status trait still says OK.
> > The status trait will now remain OK forever or until we start
> > collecting availability again.
> > 
> > 
> > 2a) Starting up the deployment  (trait checking faster than
> > availability)
> > ---------------------------------
> > I leave the deployment with its default availability collection
> > period of 10mins but change its Status trait to be collected every
> > 30seconds.
> > Deployment starts off red and status = DOWN
> > Execute the enable operation. It completes successfully very
> > quickly.
> > Wait 1min
> > The status trait says DOWN and the availability still says red, but
> > the app is actually available.
> > Wait some more time, possibly upto 10mins.
> > The availability will now show green. Shortly after that the status
> > will go to OK
> > 
> > 
> > 2b)  Starting up the deployment (trait checking slower than
> > availability)
> > ---------------------------------------
> > I leave the deployment with its default availability collection
> > period of 10mins but change its Status trait to be collected every
> > 20minutes
> > Deployment starts off red and status = DOWN
> > Execute the enable operation. It completes successfully very
> > quickly.
> > Wait 1min
> > The status trait says DOWN and the availability still says red, but
> > the app is actually available.
> > Wait some more time, possibly upto 10mins.
> > The availability will now show green, but (assuming the trait
> > collection still hasn't fired) the status trait still says DOWN.
> > Wait some more time, possibly upto another 10mins.
> > The status trait will eventually flip to OK
> > 
> > 
> > Summary and thoughts:
> > 
> > i) IMHO its not very intuitive that having various different
> > schedules for availability and trait collection can result in
> > radically different results. I think the general idea around
> > metric/avail/traits should be that if the system ends up in a
> > steady
> > state that the metric/avail/trait should eventually get into that
> > same steady state. The frequency of collection should just
> > determine
> > how quickly its get there.
> > So looking at the results above I see the following:
> > 
> > 2b) is intuitive, slow trait schedule => get to steady state slowly
> > 1a) is intuitive, fast trait schedule => get to steady state
> > quickly
> > 2a) is not intuitive, fast trait schedule doesn't update the trait
> > quickly
> > 1b) looks broken, trait may never get into correct state
> > 
> > I'm not sure what could be done about the last two. 1b) could
> > possibly be mitigated by letting metrics and traits be collected
> > one
> > final time after the avail has gone red. 2a) could perhaps be
> > addressed to a limited extent by updating what happens when an
> > operation executes, as discussed below.
> > 
> > 
> > ii) For operations launched from the RHQ CLI/UI there should be a
> > way
> > (if there isnt already) to trigger an availability/trait/metric
> > collection after the operation has completed. [If we already have
> > this, then the AS7 plugin needs to be updated to support it]
> > That way users will see the very latest state as soon as possible
> > after the operation is done. Its obviously fine for out of band
> > events, e.g. someone manually deleting the application from the
> > filesystem, to get picked up by the scheduled polling which the
> > plugin does.
> > 
> > 
> > What does everyone think? BTW I'm not arguing about why we've ended
> > up where we have, e.g. not collecting metrics/traits for down
> > resources makes perfect sense from a performance perspective. I
> > just
> > want us to think about how we can improve going forward.
> > 
> > Cheers
> > Charles
> > 
> > 
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/rhq-devel
> > 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel
> 


More information about the rhq-devel mailing list