general RHQ feedback and roadmap

Andreas Dietrich adi at aspicon.de
Wed Nov 10 14:29:21 UTC 2010


  Hi Joseph,

thanks for your interest on this point.
I'll respond below and may post more time consuming or detailed answers 
later.

Regards
Andreas :-)


On 11/10/2010 05:42 AM, Joseph Marques wrote:
> Andreas, thanks for the detailed feedback.  Responses in line below...
>
> n 10/29/2010 05:46 AM, Andreas Dietrich wrote:
>> ...
>> Benefits:
>>
>> - we can easily rollout RHQ agents to our platforms (Linux, Windows, AIX, HP-UX, ...) and it basically works (adjustments on JVM params necessary to stabelize agents on AIX, HP-UX)
> Can you characterize, explain, or copy/paste the tweaks you found
> necessary?  It might necessitate a change to the default scripts, or at
> the very least be useful input to the FAQ and/or installation guide.

- agent crashes and related JVM parameter fixes (has to do with the 
Classloaders and special garbage collection handling for classes):

<rhq-agent-dir>/bin/rhq-agent-env.sh:
# Sun JVM (/non-IBM):
RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-XX:+UseConcMarkSweepGC 
-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled"
# IBM JVM (AIX):
#RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Xgcpolicy:optavgpause -Xalwaysclassgc"


>> - the setup of all the agents and the server takes some time, but it's not complicated
> And in case you haven't seen it --
> http://www.rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentInstallation-AutoUpdatingTheAgent
>
> Also, we'd be interested in your feedback around --
> http://management-platform.blogspot.com/2010/08/remotely-installing-agent.html
>

We tried the first one but I did not know of the second (anyways it 
won't fit till it does not allow to rollout differently configured agents)

With the auto-agent-update we had issues, but maybe we did not spend 
enough time to resolve those:

- depending on the machine we use some <rhq-agent-dir>/jre (symbolic) 
link or like/have to move the JRE itself to this place
=> it's not copied after the update
- we have some individually filled folder <rhq-agent-dir>/scripts that 
is not copied after the update

=> so we skipped this feature for now

>> - we could rather easily integrate our own agent plugins (Oracle db monitoring, log file monitoring)
>>
>> - we can setup alarms for our resources and manage them hierachically, with some useful bulk functionality (e.g. groups, templates, individual read-only rules)
>>
>> - we can investigate issues based on detailed, aggregated and graphically represented data (enough and efficient for the monitoring itself and to track down potential problematic resources)
>>
>> - there are much more, but the above is the essential part
>>
>>
>> Situation without our own modifications/extensions (most important first):
>>
>> - managing up to 100 alerts per day is cumbersome (especially for maintenance situation on networks/tunnels and machines)
> Do you have any ideas for how to help alleviate this?
I will provide you with a WYSIWYG explanation of the solution we have in 
place right now that works quite well.
But this would take quite some time to anonymize ... I am sorry I don't 
have the time right now to post it.
The abstracted idea is presented below.


>> ... the currently implemented screens and fitler pages are insufficient for this if you have a lot of resources and machines where alerts can't be ignored and deleted easily
>> ( remember that one has to keep track of problems and is responsible for noticing details that might lead to future problems ... especially if it is on a b2b contract basis)
>>
>> ... a lot of this maintenance work is an essential part of our daily work and our customers and not so exceptional!
>>
>> ... we need auto-acknowledgement and deletion rules (combinable of customer / machine-regexp / resource-regexp / resource-type-regexp / resource-type-natkey*-regexp / resource-natkey*-regexp / alert-condition-type [AVAILABILITY|MEASUREMENT] / individually setable acknowledgement-status ["false", "recovered", "ignore", "todo", "maintenance", ... ] / start-time / end-time )
>> * natkey = natural key, e.g. "Platforms - Linux - CPU 0" or "mach1.company1.com - CPU 0"
> Today, alert history can be viewed a single resource at a time, a group
> at a time, or for any resource visible to the user via the alert history
> subsystem view.  Filters vary between the screen but generally have
> time/date, priority (low,medium,high), type (measurement-based
> condition, operation-based condition, etc).
>
As I said, the filters are not sufficient if you have some houndred alarms.
Also it does not provide some hierarchical overview, ideally colored, 
e.g. (similar to what we have in place):

# unacknowledged alerts
-- customer 1
*** machine 1.1  (last highest prio alert: "Agent went down", 32x 
alerts, 1 ... 3 days ago)
-- customer 2
*** machine 2.3  (last highest prio alert: "User load warning", 1x 
alerts, 0 ... 0 days ago)

# recovered alerts
-- customer 1
*** machine 1.1 (...)
*** machine 1.2 (...)

# todo-known-issue-#123 alerts
-- customer 5
*** machine 5.3 (...)

# ignore recovered dependent alerts (e.g.
-- customer 1
*** machine 1.1  (last highest prio alert: "DB Instance went down", 32x 
alerts, 1 ... 3 days ago)
.....

> In the future, it's likely that we would extend the new search facility
> in RHQ to alerts --
> http://josephmarques.wordpress.com/2010/08/04/rhqs-powerful-new-search-facility/
>
> Implementing the search bar contract for alerts would mean that you
> could then have saved searches for alert history, to help manage that
> daily flow.
>
I don't know if we need saved searches, but beeing more flexible in 
searching alerts and alert definitions would be a big benefit!
Encoding the filters in the URL would be great. So you can easily 
bookmark, post or otherwise reuse filters.

> As for the individually settable acknowledgement-status, I've filed this
> enhancement request for you --
> https://bugzilla.redhat.com/show_bug.cgi?id=651504
Thanks.

>> -- all this must be managable in an easy to use (click- and scoll-optimized control view / dt. "Leitstand")  (aggregated data displayed with meaningful colors, tooltips etc.)
>> (I will provide some anonymized screenshot if I have time to develop it so you have an idea how this looks like)
>>
>> ... imagine monitoring personal is informed via some notifications, now it needs to filter out the most critical customer/machine/database/resource that needs attention
>> ... furthermore we need to fastly categorize or delete alerts so another person or myself do not have to deal with them again (or sees why it's there, who will work on it, what is planned todo there, how further similar alarms should be treated, ...)
>>
>> - compatible groups do not inherit alert templates which makes them less useful
>> ...e.g. a general "Log ERROR found" without some notification setup can't be modified for DB alerts "DB alert logs" group (e.g. we would like to get an email for this one)
> Alert templates and group alert definitions are both convenience
> mechanisms.  If you create an alert template, it will clone that
> definition to all resources of that type.  If you create a group alert
> definition, it will clone that definition to all members in that group.
>
I know.
> I'm a bit unsure of what you mean by "do not inherit alert templates."
> Are you saying you want to be able to create alert template 1 for type
> A, then to a compatible group of type A, and edit template 1 just for
> the members in the group?  Or that you want to be able to create a new
> group alert definition by starting from the current state of one of the
> alert templates?  Please provide your thoughts on what features you'd
> like to see in this area to make alerting more useful.
Both. But I understand that this is no easy thing to do. For explanation 
a shortened(!) real-world example may be useful...
_____________________________________________

(we have some own Logreader plugin:)

RT1.AD1: resource type alert definition/template: "Log ERROR found", no 
notifications setup  =>  for any logfile setup or found via discovery 
this should be the default
(we consider log errors in general not worth informing somebody via 
mail, but would like to see it in e.g. the alert history)

RT1.CRG1.AD1: compatible resource (dyna-)group (DB logs: "alert_*.log") 
alert definition/template: "DB Log ERROR found", email notification setup


We can setup the above with the following results:

S1) Solution 1 (works in RHQ 3.0.0):
- R1: some resource "/db/logs/alert_oradb.log" would inherit the 
following alert definitions:
-- RT1.AD1.R1: "Log ERROR found"
-- RT1.CRG1.AD1.R1: "DB Log ERROR found"

=> we would have to disable RT1.CRG1.AD1.R* for all such resources R* to 
not be notified twice (cumbersome and inefficient, especially on RT1.AD1 
or RT1.CRG1.AD1 updates these would be reset if RT1.AD1.R* are not 
read-only)

S2) Solution 2 (does not currently work in RHQ 3.0.0):
- RT1.CRG1.AD1 does not show/inherit RT1.AD1
-- we can't modify "Log ERROR found" here, having to setup S1
--- rename "Log ERROR found" to "DB Log ERROR found"
--- setup email notification for "MailAlert" role members


If we would have S2 in place some problems arise:

P1) what if somebody updates RT1.AD1: adding some email notification for 
"xyz at mycompany.com"
-- how should RT1.CRG1 be updated? (e.g. add notification to existing ones?)

P2) what if we would have another group RT1.CRG2 (speculative example: 
"*oradb*.log" logs)
- RT1.CRG2.AD1: we would like to rename "Log ERROR found" to "OraDB log 
ERROR found" and setup some Mobicent/Call notification (+ mail notification)

=> which templates should R1 inherit or not? (all: RT1.CRG1.AD1, 
RT1.CRG2.AD1, maybe the most specific RT1.CRG2.AD1, ...)

_____________________________________________

All this is a bit complex and would need some useful concept (C).
Additionally it could make sense for us to have (compatible) groups 
consisting of groups (and resources).

In my view the whole thing could break down to this:


- you should remove the existing "alert templates" feature and 
substitute it by utilizing the group feature
(alert templates are basically non-deletable compatible groups including 
all types of a resource) => this should make things easier for users and 
developers)

- groups should inherit all ADs (and thus ANs (alert notifications)) 
from other groups including the same resources (and maybe more) (= 
ancestor groups)

- the user should be able to disable (configured and inherited) group 
ADs and/or ANs (e.g. -AD1, +AD2, -AD2.AN1, +AD3.AN2, ...)  (-/+ for 
disable/enable)

- do not allow inherited ADs to be deleted (you could disable them if 
you like, but are able to reenable them if wanted)
(the deletion is currently possible in RHQ 3.0.0 for resources, but it 
does not make sense to me and is actually a bad thing, don't you think so?)

- if the group memberships changes (on a manual update or an automated 
group definition update) the inheritance rules must be reapplied 
internally and only now not anymore inherited ADs must be removed from 
groups and resources
(e.g. a resource is taken out from some group RT1.CRG1 and thus it is 
not anymore an ancestor for some group RT1.CRG2)
we have to make sure that the ADs/ANs that have been disabled/enabled 
before and will stay the same if still inherited

- name and description of inherited ADs should be modifiable (see S2 
from above for a production use case)

>> - we can't auto-setup customer groups (the group definitions do not allow "and/or" conditions
>> ... you can't say something like 'machine-*.area1.company1.com' and 'machine-*.area2.company1.com' belong to group 'company1'
> Agreed, and group definitions will eventually be expanded to include
> more robust condition rules.  In the meantime, would
> "resource.name.endswith = company1.com" work?
Then we would have no problem ;-) (For most customers this works, but 
not for all)
Groups of groups would also solve this problem as described above.
>> - customers setup for RHQ usage (setup with view only rights on their group) can see all other users available
> This is working as intended.  One of the primary use cases is found in
> the alerting subsystem.  When you create an alert definition and want to
> set up notifications, you need to choose from valid users and roles in
> the system.  This is why all users have visibility into that type of data.
The intention should be extended then :-)
We do not want to bother customer X user X.1 with some alert "User Load 
warning on some machine from customer Y" because user Z.1 thinks this is 
funny spam ;-)
(Besides: we would not let customer users setup anything on our system)
And we do not want to let user X.1 from customer X see all users data 
(email, ...) Y.* from customer Y :-)

>> - of course there is more, but the above again are the more important to us (if I did not forget something ... I am in a hurry since I have a home office day and have to change the diapers of my son *puh* ;-))
>>
>>
>> I could write some more pages about this, but I think the alarm and resource management is the most important to work on for the current stage of the project.
>> I would leave e.g. the ongoing graphical enhancements (GWT, graph types) aside and concentrate on the essential and much more important features if it would be for our usage scenarios.
>> Of course I understand that a "fancy" looking project sells much better especially to managers, but if this is not an important point on the "roadmap" I would skip it for now and concentrate on the user that "does not like to use RHQ if possible". Because RHQ is only there to notice, track down and recover problematic resources", RHQ-setup and -cleanup are necessary evils that should be kept minimal. The real work (~95% of the time) should be spent on the problematic resource or some related one itself ... db setup/recovery, router, firewalls, os setup, application changes/restarts etc.
>>
>> Kind regards
>> Andreas :-)
>>
>>
>> PS: Currently our own management overview is heavily based on Eclipse BIRT reports based itself on RHQ description XML tags (in description of inventories, alert definitions, groups), Oracle DB Views and Table Functions working over the RHQ DB. This is rather easy to implement and fast. We plan to extend it to utilize the RHQ client API to not only visualize the data, but modify it. Of course this is not nice since we should build this into RHQ itself. But it's easier to start with some DB schema you know than with the big RHQ monolith itself (We know hibernate, Struts, ADF, JSF and alike, but it's still hard to say for us how much time has to be spend then).
>> _______________________________________________
>> rhq-devel mailing list
>> rhq-devel at lists.fedorahosted.org
>> https://fedorahosted.org/mailman/listinfo/rhq-devel
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel


More information about the rhq-devel mailing list