Hi Joseph,
thanks for your interest on this point.
I'll respond below and may post more time consuming or detailed answers
later.
Regards
Andreas :-)
On 11/10/2010 05:42 AM, Joseph Marques wrote:
Andreas, thanks for the detailed feedback. Responses in line
below...
n 10/29/2010 05:46 AM, Andreas Dietrich wrote:
> ...
> Benefits:
>
> - we can easily rollout RHQ agents to our platforms (Linux, Windows, AIX, HP-UX, ...)
and it basically works (adjustments on JVM params necessary to stabelize agents on AIX,
HP-UX)
Can you characterize, explain, or copy/paste the tweaks you found
necessary? It might necessitate a change to the default scripts, or at
the very least be useful input to the FAQ and/or installation guide.
- agent crashes and related JVM parameter fixes (has to do with the
Classloaders and special garbage collection handling for classes):
<rhq-agent-dir>/bin/rhq-agent-env.sh:
# Sun JVM (/non-IBM):
RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-XX:+UseConcMarkSweepGC
-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled"
# IBM JVM (AIX):
#RHQ_AGENT_ADDITIONAL_JAVA_OPTS="-Xgcpolicy:optavgpause -Xalwaysclassgc"
> - the setup of all the agents and the server takes some time, but
it's not complicated
And in case you haven't seen it --
http://www.rhq-project.org/display/JOPR2/RHQ+Agent+Installation#RHQAgentI...
Also, we'd be interested in your feedback around --
http://management-platform.blogspot.com/2010/08/remotely-installing-agent...
We tried the first one but I did not know of the second (anyways it
won't fit till it does not allow to rollout differently configured agents)
With the auto-agent-update we had issues, but maybe we did not spend
enough time to resolve those:
- depending on the machine we use some <rhq-agent-dir>/jre (symbolic)
link or like/have to move the JRE itself to this place
=> it's not copied after the update
- we have some individually filled folder <rhq-agent-dir>/scripts that
is not copied after the update
=> so we skipped this feature for now
> - we could rather easily integrate our own agent plugins (Oracle
db monitoring, log file monitoring)
>
> - we can setup alarms for our resources and manage them hierachically, with some
useful bulk functionality (e.g. groups, templates, individual read-only rules)
>
> - we can investigate issues based on detailed, aggregated and graphically represented
data (enough and efficient for the monitoring itself and to track down potential
problematic resources)
>
> - there are much more, but the above is the essential part
>
>
> Situation without our own modifications/extensions (most important first):
>
> - managing up to 100 alerts per day is cumbersome (especially for maintenance
situation on networks/tunnels and machines)
Do you have any ideas for how to help alleviate this?
I will provide you with a
WYSIWYG explanation of the solution we have in
place right now that works quite well.
But this would take quite some time to anonymize ... I am sorry I don't
have the time right now to post it.
The abstracted idea is presented below.
> ... the currently implemented screens and fitler pages are
insufficient for this if you have a lot of resources and machines where alerts can't
be ignored and deleted easily
> ( remember that one has to keep track of problems and is responsible for noticing
details that might lead to future problems ... especially if it is on a b2b contract
basis)
>
> ... a lot of this maintenance work is an essential part of our daily work and our
customers and not so exceptional!
>
> ... we need auto-acknowledgement and deletion rules (combinable of customer /
machine-regexp / resource-regexp / resource-type-regexp / resource-type-natkey*-regexp /
resource-natkey*-regexp / alert-condition-type [AVAILABILITY|MEASUREMENT] / individually
setable acknowledgement-status ["false", "recovered",
"ignore", "todo", "maintenance", ... ] / start-time /
end-time )
> * natkey = natural key, e.g. "Platforms - Linux - CPU 0" or
"mach1.company1.com - CPU 0"
Today, alert history can be viewed a single resource at a time, a group
at a time, or for any resource visible to the user via the alert history
subsystem view. Filters vary between the screen but generally have
time/date, priority (low,medium,high), type (measurement-based
condition, operation-based condition, etc).
As I said, the filters are not sufficient if you have some houndred alarms.
Also it does not provide some hierarchical overview, ideally colored,
e.g. (similar to what we have in place):
# unacknowledged alerts
-- customer 1
*** machine 1.1 (last highest prio alert: "Agent went down", 32x
alerts, 1 ... 3 days ago)
-- customer 2
*** machine 2.3 (last highest prio alert: "User load warning", 1x
alerts, 0 ... 0 days ago)
# recovered alerts
-- customer 1
*** machine 1.1 (...)
*** machine 1.2 (...)
# todo-known-issue-#123 alerts
-- customer 5
*** machine 5.3 (...)
# ignore recovered dependent alerts (e.g.
-- customer 1
*** machine 1.1 (last highest prio alert: "DB Instance went down", 32x
alerts, 1 ... 3 days ago)
.....
In the future, it's likely that we would extend the new search
facility
in RHQ to alerts --
http://josephmarques.wordpress.com/2010/08/04/rhqs-powerful-new-search-fa...
Implementing the search bar contract for alerts would mean that you
could then have saved searches for alert history, to help manage that
daily flow.
I don't know if we need saved searches, but beeing more flexible in
searching alerts and alert definitions would be a big benefit!
Encoding the filters in the URL would be great. So you can easily
bookmark, post or otherwise reuse filters.
As for the individually settable acknowledgement-status, I've
filed this
enhancement request for you --
https://bugzilla.redhat.com/show_bug.cgi?id=651504 Thanks.
> -- all this must be managable in an easy to use (click- and
scoll-optimized control view / dt. "Leitstand") (aggregated data displayed with
meaningful colors, tooltips etc.)
> (I will provide some anonymized screenshot if I have time to develop it so you have
an idea how this looks like)
>
> ... imagine monitoring personal is informed via some notifications, now it needs to
filter out the most critical customer/machine/database/resource that needs attention
> ... furthermore we need to fastly categorize or delete alerts so another person or
myself do not have to deal with them again (or sees why it's there, who will work on
it, what is planned todo there, how further similar alarms should be treated, ...)
>
> - compatible groups do not inherit alert templates which makes them less useful
> ...e.g. a general "Log ERROR found" without some notification setup
can't be modified for DB alerts "DB alert logs" group (e.g. we would like to
get an email for this one)
Alert templates and group alert definitions are both convenience
mechanisms. If you create an alert template, it will clone that
definition to all resources of that type. If you create a group alert
definition, it will clone that definition to all members in that group.
I know.
I'm a bit unsure of what you mean by "do not inherit alert
templates."
Are you saying you want to be able to create alert template 1 for type
A, then to a compatible group of type A, and edit template 1 just for
the members in the group? Or that you want to be able to create a new
group alert definition by starting from the current state of one of the
alert templates? Please provide your thoughts on what features you'd
like to see in this area to make alerting more useful.
Both. But I understand that
this is no easy thing to do. For explanation
a shortened(!) real-world example may be useful...
_____________________________________________
(we have some own Logreader plugin:)
RT1.AD1: resource type alert definition/template: "Log ERROR found", no
notifications setup => for any logfile setup or found via discovery
this should be the default
(we consider log errors in general not worth informing somebody via
mail, but would like to see it in e.g. the alert history)
RT1.CRG1.AD1: compatible resource (dyna-)group (DB logs: "alert_*.log")
alert definition/template: "DB Log ERROR found", email notification setup
We can setup the above with the following results:
S1) Solution 1 (works in RHQ 3.0.0):
- R1: some resource "/db/logs/alert_oradb.log" would inherit the
following alert definitions:
-- RT1.AD1.R1: "Log ERROR found"
-- RT1.CRG1.AD1.R1: "DB Log ERROR found"
=> we would have to disable RT1.CRG1.AD1.R* for all such resources R* to
not be notified twice (cumbersome and inefficient, especially on RT1.AD1
or RT1.CRG1.AD1 updates these would be reset if RT1.AD1.R* are not
read-only)
S2) Solution 2 (does not currently work in RHQ 3.0.0):
- RT1.CRG1.AD1 does not show/inherit RT1.AD1
-- we can't modify "Log ERROR found" here, having to setup S1
--- rename "Log ERROR found" to "DB Log ERROR found"
--- setup email notification for "MailAlert" role members
If we would have S2 in place some problems arise:
P1) what if somebody updates RT1.AD1: adding some email notification for
"xyz(a)mycompany.com"
-- how should RT1.CRG1 be updated? (e.g. add notification to existing ones?)
P2) what if we would have another group RT1.CRG2 (speculative example:
"*oradb*.log" logs)
- RT1.CRG2.AD1: we would like to rename "Log ERROR found" to "OraDB log
ERROR found" and setup some Mobicent/Call notification (+ mail notification)
=> which templates should R1 inherit or not? (all: RT1.CRG1.AD1,
RT1.CRG2.AD1, maybe the most specific RT1.CRG2.AD1, ...)
_____________________________________________
All this is a bit complex and would need some useful concept (C).
Additionally it could make sense for us to have (compatible) groups
consisting of groups (and resources).
In my view the whole thing could break down to this:
- you should remove the existing "alert templates" feature and
substitute it by utilizing the group feature
(alert templates are basically non-deletable compatible groups including
all types of a resource) => this should make things easier for users and
developers)
- groups should inherit all ADs (and thus ANs (alert notifications))
from other groups including the same resources (and maybe more) (=
ancestor groups)
- the user should be able to disable (configured and inherited) group
ADs and/or ANs (e.g. -AD1, +AD2, -AD2.AN1, +AD3.AN2, ...) (-/+ for
disable/enable)
- do not allow inherited ADs to be deleted (you could disable them if
you like, but are able to reenable them if wanted)
(the deletion is currently possible in RHQ 3.0.0 for resources, but it
does not make sense to me and is actually a bad thing, don't you think so?)
- if the group memberships changes (on a manual update or an automated
group definition update) the inheritance rules must be reapplied
internally and only now not anymore inherited ADs must be removed from
groups and resources
(e.g. a resource is taken out from some group RT1.CRG1 and thus it is
not anymore an ancestor for some group RT1.CRG2)
we have to make sure that the ADs/ANs that have been disabled/enabled
before and will stay the same if still inherited
- name and description of inherited ADs should be modifiable (see S2
from above for a production use case)
> - we can't auto-setup customer groups (the group definitions
do not allow "and/or" conditions
> ... you can't say something like 'machine-*.area1.company1.com' and
'machine-*.area2.company1.com' belong to group 'company1'
Agreed, and group definitions will eventually be expanded to include
more robust condition rules. In the meantime, would
"resource.name.endswith = company1.com" work?
Then we would have no
problem ;-) (For most customers this works, but
not for all)
Groups of groups would also solve this problem as described above.
> - customers setup for RHQ usage (setup with view only rights on
their group) can see all other users available
This is working as intended. One of the primary use cases is found in
the alerting subsystem. When you create an alert definition and want to
set up notifications, you need to choose from valid users and roles in
the system. This is why all users have visibility into that type of data.
The
intention should be extended then :-)
We do not want to bother customer X user X.1 with some alert "User Load
warning on some machine from customer Y" because user Z.1 thinks this is
funny spam ;-)
(Besides: we would not let customer users setup anything on our system)
And we do not want to let user X.1 from customer X see all users data
(email, ...) Y.* from customer Y :-)
> - of course there is more, but the above again are the more
important to us (if I did not forget something ... I am in a hurry since I have a home
office day and have to change the diapers of my son *puh* ;-))
>
>
> I could write some more pages about this, but I think the alarm and resource
management is the most important to work on for the current stage of the project.
> I would leave e.g. the ongoing graphical enhancements (GWT, graph types) aside and
concentrate on the essential and much more important features if it would be for our usage
scenarios.
> Of course I understand that a "fancy" looking project sells much better
especially to managers, but if this is not an important point on the "roadmap" I
would skip it for now and concentrate on the user that "does not like to use RHQ if
possible". Because RHQ is only there to notice, track down and recover problematic
resources", RHQ-setup and -cleanup are necessary evils that should be kept minimal.
The real work (~95% of the time) should be spent on the problematic resource or some
related one itself ... db setup/recovery, router, firewalls, os setup, application
changes/restarts etc.
>
> Kind regards
> Andreas :-)
>
>
> PS: Currently our own management overview is heavily based on Eclipse BIRT reports
based itself on RHQ description XML tags (in description of inventories, alert
definitions, groups), Oracle DB Views and Table Functions working over the RHQ DB. This is
rather easy to implement and fast. We plan to extend it to utilize the RHQ client API to
not only visualize the data, but modify it. Of course this is not nice since we should
build this into RHQ itself. But it's easier to start with some DB schema you know than
with the big RHQ monolith itself (We know hibernate, Struts, ADF, JSF and alike, but
it's still hard to say for us how much time has to be spend then).
> _______________________________________________
> rhq-devel mailing list
> rhq-devel(a)lists.fedorahosted.org
>
https://fedorahosted.org/mailman/listinfo/rhq-devel
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/rhq-devel