Questions around traits/availability/operations

Charles Crouch ccrouch at redhat.com
Mon Jul 9 13:59:55 UTC 2012



----- Original Message -----
> > not collecting metrics/traits for down
> > resources makes perfect sense from a performance perspective.
> 
> Why assume that we could ever collect a metric or trait value from a
> resource that is down? In general, if a resource is down, we can't
> query it to know anything about it.

That depends.
Often for container based resources, we're actually querying the container for information about its children, so 

> Now, in your particular case,
> the trait (status) isn't really a value that is retrieved from the
> resource, so you could technically ask the plugin for it and you
> could get a valid value back, but that is a very specific
> implementation of that trait - why think in the general case that
> this is always true? If a resource is down, in general, assume you
> can know nothing about it since it won't be up to respond to the
> plugin's queries about its state.

I'm advocating that we provide better support for the scenario I described.
You are right, we need to come up with a mechanism which supports all the various types of plugins we have.
> 
> What we could do is add a new piece of metadata to <metric> - that
> being something like "collectEvenIfResourceIsDown=true" - if true, I
> guess we might be able to change the MeasurementManager to make the
> call to collect that metric even if the resource is down.
> 
> ----- Original Message -----
> > I didn't read this whole thing yet but... as of RHQ today:
> > 
> > "its default availability collection period of 10mins"
> > 
> > is incorrect. Our default avail collection period is 1 minute now.
> > 
> > ----- Original Message -----
> > > Say I have a resource with a Enable/Disable operation that also
> > > has
> > > a
> > > Status trait, e.g. a Deployment/application service under AS7.
> > > Here are some scenarios that I've tried out (apologies if I've
> > > mis-remembered something here, I've not run the through the whole
> > > battery of scenarios after my initial tests, but I think the
> > > following are accurate)
> > > 
> > > 
> > > 1a) Shutting down the deployment (trait checking faster than
> > > availability)
> > > ---------------------------------------
> > > I leave the deployment with its default availability collection
> > > period of 10mins but change its Status trait to be collected
> > > every
> > > 30seconds.
> > > Deployment starts off green and status = OK
> > > Execute the disable operation. It completes successfully very
> > > quickly.
> > > Wait 1min
> > > The status trait now says DOWN, but the availability still says
> > > green, but the app is actually not available.
> > > Wait some more time, possibly upto 10mins.
> > > The availability will now show red.
> > > 
> > > 
> > > 1b)  Shutting down the deployment (trait checking slower than
> > > availability)
> > > ---------------------------------------
> > > I leave the deployment with its default availability collection
> > > period of 10mins but change its Status trait to be collected
> > > every
> > > 20minutes
> > > Deployment starts off green and status = OK
> > > Execute the disable operation. It completes successfully very
> > > quickly.
> > > Wait 1min
> > > The status trait says OK and the availability still says green,
> > > but
> > > the app is actually not available.
> > > Wait some more time, possibly upto 10mins.
> > > The availability will now show red, but (assuming the trait
> > > collection still hasn't fired) the status trait still says OK.
> > > The status trait will now remain OK forever or until we start
> > > collecting availability again.
> > > 
> > > 
> > > 2a) Starting up the deployment  (trait checking faster than
> > > availability)
> > > ---------------------------------
> > > I leave the deployment with its default availability collection
> > > period of 10mins but change its Status trait to be collected
> > > every
> > > 30seconds.
> > > Deployment starts off red and status = DOWN
> > > Execute the enable operation. It completes successfully very
> > > quickly.
> > > Wait 1min
> > > The status trait says DOWN and the availability still says red,
> > > but
> > > the app is actually available.
> > > Wait some more time, possibly upto 10mins.
> > > The availability will now show green. Shortly after that the
> > > status
> > > will go to OK
> > > 
> > > 
> > > 2b)  Starting up the deployment (trait checking slower than
> > > availability)
> > > ---------------------------------------
> > > I leave the deployment with its default availability collection
> > > period of 10mins but change its Status trait to be collected
> > > every
> > > 20minutes
> > > Deployment starts off red and status = DOWN
> > > Execute the enable operation. It completes successfully very
> > > quickly.
> > > Wait 1min
> > > The status trait says DOWN and the availability still says red,
> > > but
> > > the app is actually available.
> > > Wait some more time, possibly upto 10mins.
> > > The availability will now show green, but (assuming the trait
> > > collection still hasn't fired) the status trait still says DOWN.
> > > Wait some more time, possibly upto another 10mins.
> > > The status trait will eventually flip to OK
> > > 
> > > 
> > > Summary and thoughts:
> > > 
> > > i) IMHO its not very intuitive that having various different
> > > schedules for availability and trait collection can result in
> > > radically different results. I think the general idea around
> > > metric/avail/traits should be that if the system ends up in a
> > > steady
> > > state that the metric/avail/trait should eventually get into that
> > > same steady state. The frequency of collection should just
> > > determine
> > > how quickly its get there.
> > > So looking at the results above I see the following:
> > > 
> > > 2b) is intuitive, slow trait schedule => get to steady state
> > > slowly
> > > 1a) is intuitive, fast trait schedule => get to steady state
> > > quickly
> > > 2a) is not intuitive, fast trait schedule doesn't update the
> > > trait
> > > quickly
> > > 1b) looks broken, trait may never get into correct state
> > > 
> > > I'm not sure what could be done about the last two. 1b) could
> > > possibly be mitigated by letting metrics and traits be collected
> > > one
> > > final time after the avail has gone red. 2a) could perhaps be
> > > addressed to a limited extent by updating what happens when an
> > > operation executes, as discussed below.
> > > 
> > > 
> > > ii) For operations launched from the RHQ CLI/UI there should be a
> > > way
> > > (if there isnt already) to trigger an availability/trait/metric
> > > collection after the operation has completed. [If we already have
> > > this, then the AS7 plugin needs to be updated to support it]
> > > That way users will see the very latest state as soon as possible
> > > after the operation is done. Its obviously fine for out of band
> > > events, e.g. someone manually deleting the application from the
> > > filesystem, to get picked up by the scheduled polling which the
> > > plugin does.
> > > 
> > > 
> > > What does everyone think? BTW I'm not arguing about why we've
> > > ended
> > > up where we have, e.g. not collecting metrics/traits for down
> > > resources makes perfect sense from a performance perspective. I
> > > just
> > > want us to think about how we can improve going forward.
> > > 
> > > Cheers
> > > Charles
> > > 
> > > 
> > > _______________________________________________
> > > rhq-devel mailing list
> > > rhq-devel at lists.fedorahosted.org
> > > https://fedorahosted.org/mailman/listinfo/rhq-devel
> > > 
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/rhq-devel
> > 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel
> 


More information about the rhq-devel mailing list