Fwd: New RHQ Groups Summary/Activity pages.. thoughts?

Andreas Dietrich adi at aspicon.de
Tue Feb 15 08:43:54 UTC 2011


  Added some comment/link to the input from Steve about some "operator 
summary view", generally called "Control Center" (german "Leitstand") 
below...

Kind regards
Andreas :-)

On 02/14/2011 12:13 PM, Heiko W.Rupp wrote:
> Very good feedback from Steve Millidge:
>
> Anfang der weitergeleiteten E-Mail:
>
>> Von: "Steve Millidge"<smillidge at c2b2.co.uk>
>> Datum: 13. Februar 2011 14:50:06 MEZ
>> An: "'Heiko W.Rupp'"<hrupp at redhat.com>
>> Betreff: RE: New RHQ Groups Summary/Activity pages.. thoughts?
>>
>> Heiko,
>>
>> Some initial thoughts from experience out in the field. Apologies for the
>> length!
>>
>> The Summary pages are one of the best features of RHQ and the page that
>> delivers the most benefit to customers. I was a bit sad to see it disappear
>> from the "first tab" in RHQ 4 when you click on a resource.  One great
>> feature that could be added on the Resource summary screen is a rollup of
>> all child resources alerts, OOB metrics, operations and config changes. For
>> example it would be great when clicking on a JBoss AS to see that an alert
>> had been generated from a JMS queue hosted by the server. If this was
>> carried out throughout the whole resource tree you could easily see by
>> clicking on a Platform any alerts generated by any servers/services on that
>> platform. As it is at the moment you need to have a good idea what the
>> problem may be to drill down to the correct resource level and check
>> metrics, config changes and alerts etc.
>>
>> Addition of group summaries would be a great addition and is better than the
>> current group page which tends to just show availability. A key thing for a
>> group summary view would be to quickly draw an operator's attention to
>> anomalies within the group.
>>
>> One thing to add at a group level would be a traffic light for availability
>> which can be drilled down. Green for all members up, Amber for some are down
>> some are up, Red for all resources down. You can then drill down for exact
>> details of each group member.
>>
>> Recent metrics at group level as the sparklines. I'm not sure how much value
>> this will add as the range bars may be a bit wide. Perhaps a moving average
>> value with some indication of the range of values within the group could be
>> more useful.
>>
>> Alerts, Config changes and Operations should be shown when they were
>> generated by a group alert or operations on the whole group. I think it
>> would also be useful to show whether an operation had been carried out on an
>> individual group member but not the whole group. Especially if it failed.
>> Perhaps the traffic light concept could be used next to the operation to
>> show whether it was executed successful on all members or only some.

>> In general good summarisation is a feature lacking in RHQ at the moment. RHQ
>> does a fantastic job of gathering a load of fine grained metrics etc. at a
>> resource level and is therefore invaluable as an admin tool for very
>> technical operators. What is needed is some "top level" view that can be
>> created to give an operator a warm feeling that everything is OK and then if
>> anything anomalous occurs, the operator requires immediate visual feedback
>> and a signpost to the suffering resource. Followed by rapid drill down to
>> the detailed metrics and timeframe where the problem occurred. These
>> operators will then typically call in deep support responsible for that
>> component who can use the current functionality to get at all the
>> information they require to diagnose.
>>
>> For example for one of our customers we developed a single JSP page
>> dashboard which we deployed to the RHQ server which gave a summarised view
>> of the state of their system. All servers were listed with traffic lights.
>> Key group metrics were also shown with current average and total values for
>> the group and traffic lights to indicate anomalous metrics were in the
>> group. These traffic lights then could be clicked on to show the server with
>> the anomalous value. For example number of messages on a JMS queue was a
>> critical metric. This queue is present on 34 servers so the top level
>> dashboard showed the summed count and average for all servers with a traffic
>> light if the value was out of some bounds. Clicking on the traffic light and
>> count gave a view of the queue counts on each server with a traffic light
>> next to each. Further drill down was a deep link into the RHQ page for the
>> resource (queue).
See also my comments here that refer to such a Control Center view as we 
implemented it for ourselves:

https://fedorahosted.org/pipermail/rhq-devel/2011-February/000585.html  
(starting from "Control Center View")

Depending on the amount of servers/customers(groups) it is important for 
it to

- spot everything (important) on an aggregated level fitting on an HTML 
area of let's say 1000x600 px
(if we assume an operator has at least some screen resolution of 
1024x768 and a standard browser and an OS task bar on the screen)
-- e.g. on really big environments with >> 100 servers some kind of 
color-coded pixel map can be efficient for the top-level view
-- it could make sense to break this view down in a standard "filtered 
view" and an "unfiltered view" (we have this in place)
--- the filtered view more or less only shows the more critical states 
and hides the uncritical ones
--- for the daily work the filtered view is much more useful and 
generally faster

- drill down options as some mentioned above leading to the most 
important RHQ pages

- be fast (should not take more than 30 s to load)
-- since this may heavily depend on the amount of alerts, 
unavailability, ... (and of course resources, groups, system changes in 
general, ...) there should be some "emergency" view as well
--- this is e.g. necessary if many resources have problems caused by 
some "bottleneck issues" (complete networks down, os/db/RHQ server/RHQ 
agent/RHQ plugin updates causing problems on many resources at once, ...)
--- we have this for our own and it shows only alerts, unavailability, 
... for critical resources (platforms, db instances, ping services) (we 
call it the "VIP view" :-) )

- robust (ideally it should be as independent as possible from the 
general RHQ frameworks to not suffer from performance or implementation 
problems from the rest)
-- e.g. we are only relying on the RHQ db (oracle, views, table 
functions, left outer joins that "can ignore" missing data) and running 
it on a different machine, JBoss based on Eclipse BIRT reports
--- I guess this is not a good approach for the official RHQ version 
which seems more db independent, based on JBoss, GWT, Seam, ..., but it 
definitely has some advantages that could eventually be combined somehow

- quick access to (bulk) operations, also based on groups, platforms, 
customers, ... (as we implemented it for ourselves saving us a lot of 
time where current RHQ 3.0 is not really usable)
-- deletion of alerts
-- acknowledgement of alerts, unavailability, conspicious setups (alert 
definitions)
--- this especially helps multiple working operators to quickly assign 
the above to somebody or "mark" them with a meaningful acknowledgement 
state or comment for further investigation/processing
-- uninventory of resources (e.g. if a whole resource tree had become 
unavailable because it does not exist/is not needed anymore)
-- enable/disable alert definitions (e.g. useful on maintenance scenarios)



>> On the subject of auto-refresh. I would be careful here as some of the group
>> metric calculations could be expensive to perform and therefore auto-refresh
>> may hammer the database server. I think some of the values would need to be
>> precalculated in the database if autorefresh at the group level was
>> performed. If the heavyweight nature of the database queries can be solved
>> autorefresh would be useful.
>>
>> Steve Millidge
>> Director
>> C2B2
>>
>> Providing the foundations for Enterprise Scale Java.
>>
>> T: 08450 539457
>> M: 07920 100626
>> W: www.c2b2.co.uk
>> E: smillidge at c2b2.co.uk


More information about the rhq-devel mailing list