Questions around traits/availability/operations

Jay Shaughnessy jshaughn at redhat.com
Mon Jul 9 15:26:45 UTC 2012


On 7/9/2012 9:52 AM, Charles Crouch wrote:
>
> ----- Original Message -----
>> I didn't read this whole thing yet but... as of RHQ today:
>>
>> "its default availability collection period of 10mins"
>>
>> is incorrect. Our default avail collection period is 1 minute now.
> That's wrong too I think. Don't we have different default avail schedules between platforms/servers/services?
We have different defaults.   Server resources default to 1 minute avail 
checks.  Services default to 10 minutes.

>
>> ----- Original Message -----
>>> Say I have a resource with a Enable/Disable operation that also has
>>> a
>>> Status trait, e.g. a Deployment/application service under AS7.
>>> Here are some scenarios that I've tried out (apologies if I've
>>> mis-remembered something here, I've not run the through the whole
>>> battery of scenarios after my initial tests, but I think the
>>> following are accurate)

I'm not 100% sure I understand what your saying here but from what I can 
gather the avail check is using the already collected trait value to 
return UP or DOWN.    If so I would at least suggest that avail perform 
the same retrieval as the trait collection.  That would at least give it 
a valid value every time it's checked.

And the operation code should request an avail check be performed ASAP, 
so it will be updated and reported "off-schedule" typically within a minute.

The trait update should be expected to be stale at least until it's next 
scheduled check, which can be tuned by the user to be as often as desired.

>>>
>>>
>>> 1a) Shutting down the deployment (trait checking faster than
>>> availability)
>>> ---------------------------------------
>>> I leave the deployment with its default availability collection
>>> period of 10mins but change its Status trait to be collected every
>>> 30seconds.
>>> Deployment starts off green and status = OK
>>> Execute the disable operation. It completes successfully very
>>> quickly.
>>> Wait 1min
>>> The status trait now says DOWN, but the availability still says
>>> green, but the app is actually not available.
>>> Wait some more time, possibly upto 10mins.
>>> The availability will now show red.
>>>
>>>
>>> 1b)  Shutting down the deployment (trait checking slower than
>>> availability)
>>> ---------------------------------------
>>> I leave the deployment with its default availability collection
>>> period of 10mins but change its Status trait to be collected every
>>> 20minutes
>>> Deployment starts off green and status = OK
>>> Execute the disable operation. It completes successfully very
>>> quickly.
>>> Wait 1min
>>> The status trait says OK and the availability still says green, but
>>> the app is actually not available.
>>> Wait some more time, possibly upto 10mins.
>>> The availability will now show red, but (assuming the trait
>>> collection still hasn't fired) the status trait still says OK.
>>> The status trait will now remain OK forever or until we start
>>> collecting availability again.
>>>
>>>
>>> 2a) Starting up the deployment  (trait checking faster than
>>> availability)
>>> ---------------------------------
>>> I leave the deployment with its default availability collection
>>> period of 10mins but change its Status trait to be collected every
>>> 30seconds.
>>> Deployment starts off red and status = DOWN
>>> Execute the enable operation. It completes successfully very
>>> quickly.
>>> Wait 1min
>>> The status trait says DOWN and the availability still says red, but
>>> the app is actually available.
>>> Wait some more time, possibly upto 10mins.
>>> The availability will now show green. Shortly after that the status
>>> will go to OK
>>>
>>>
>>> 2b)  Starting up the deployment (trait checking slower than
>>> availability)
>>> ---------------------------------------
>>> I leave the deployment with its default availability collection
>>> period of 10mins but change its Status trait to be collected every
>>> 20minutes
>>> Deployment starts off red and status = DOWN
>>> Execute the enable operation. It completes successfully very
>>> quickly.
>>> Wait 1min
>>> The status trait says DOWN and the availability still says red, but
>>> the app is actually available.
>>> Wait some more time, possibly upto 10mins.
>>> The availability will now show green, but (assuming the trait
>>> collection still hasn't fired) the status trait still says DOWN.
>>> Wait some more time, possibly upto another 10mins.
>>> The status trait will eventually flip to OK
>>>
>>>
>>> Summary and thoughts:
>>>
>>> i) IMHO its not very intuitive that having various different
>>> schedules for availability and trait collection can result in
>>> radically different results. I think the general idea around
>>> metric/avail/traits should be that if the system ends up in a
>>> steady
>>> state that the metric/avail/trait should eventually get into that
>>> same steady state. The frequency of collection should just
>>> determine
>>> how quickly its get there.
>>> So looking at the results above I see the following:
>>>
>>> 2b) is intuitive, slow trait schedule => get to steady state slowly
>>> 1a) is intuitive, fast trait schedule => get to steady state
>>> quickly
>>> 2a) is not intuitive, fast trait schedule doesn't update the trait
>>> quickly
>>> 1b) looks broken, trait may never get into correct state
>>>
>>> I'm not sure what could be done about the last two. 1b) could
>>> possibly be mitigated by letting metrics and traits be collected
>>> one
>>> final time after the avail has gone red. 2a) could perhaps be
>>> addressed to a limited extent by updating what happens when an
>>> operation executes, as discussed below.
>>>
>>>
>>> ii) For operations launched from the RHQ CLI/UI there should be a
>>> way
>>> (if there isnt already) to trigger an availability/trait/metric
>>> collection after the operation has completed. [If we already have
>>> this, then the AS7 plugin needs to be updated to support it]
>>> That way users will see the very latest state as soon as possible
>>> after the operation is done. Its obviously fine for out of band
>>> events, e.g. someone manually deleting the application from the
>>> filesystem, to get picked up by the scheduled polling which the
>>> plugin does.
>>>
>>>
>>> What does everyone think? BTW I'm not arguing about why we've ended
>>> up where we have, e.g. not collecting metrics/traits for down
>>> resources makes perfect sense from a performance perspective. I
>>> just
>>> want us to think about how we can improve going forward.
>>>
>>> Cheers
>>> Charles
>>>
>>>
>>> _______________________________________________
>>> rhq-devel mailing list
>>> rhq-devel at lists.fedorahosted.org
>>> https://fedorahosted.org/mailman/listinfo/rhq-devel
>>>
>> _______________________________________________
>> rhq-devel mailing list
>> rhq-devel at lists.fedorahosted.org
>> https://fedorahosted.org/mailman/listinfo/rhq-devel
>>
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel





More information about the rhq-devel mailing list