Need advice about agent plugin design

Andreas Dietrich adi at aspicon.de
Fri Jun 17 07:40:25 UTC 2011


  Hi Steven,

we will release our Logreader plugin amongst others soon. Maybe it could 
help you...

Our Logreader plugin is able to capture logs or specific lines of interests.
Its primary purpose so far is to monitor logs for errors and warnings 
and to issue an RHQ alert if they occur.

We are using it successfully in various production environments now for 
quite some time for the monitoring of all kinds of different log types: 
OS logs (/var/message/log), Oracle database alert and listener logs, 
JBoss logs, RHQ logs (agent.log, rhq-server-log4j.log), specific 
application logs and many more ...

You can configure it with help of Java regular expression lists, e.g. 
some simple ones ... ".* ERROR .*", ".* WARN .*", ".* ORA-\d+ .*".
You can also exclude certain error or warning patterns in the same way, 
e.g. for known and not to be processed loglines.
The handling for abitrary log files is quite sophisticated regarding

- rolling log files
- limits on "log/alert spam" (e.g. to report only 5 loglines per 1 min 
log check where the 4th contains a line like "... skipped 300 loglines 
due to a max loglines per check of 5")
- optimizations for very large logfiles (e.g. start at last known 
position if file has not rolled yet, regexp handling, ...)
- auto discovery of certain locations, types and patterns
- capturing of historical/old loglines if needed
- agent restarts or downtimes

It stores it as traits in the database, but currently not with a 
granularity and precision as Syslog daemons do (only discovery time of 
logline, up to 4000 characters per line). (You also see the trait data 
in the GUI with up to 256 characters per line in a chronological order)

More details will be available on our homepage within the next weeks. 
But if you don't want to wait we can give you the plugin to try it out 
yourself earlier.

Kind regards
Andreas :-)


On 06/17/2011 01:01 AM, Steven North wrote:
> John,
>
> Nice to talk with you and thanks for the response.
>
> Please see below...
>
> On 6/16/2011 2:23 PM, John Mazzitelli wrote:
>> OK, Steve, sorry for not answering this earlier. The reason is because I
>> don't think I have good answers for them :) Either that or I don't have
>> a good grasp of what you need.
>>
>> On 05/29/2011 09:13 PM, Steven North wrote:
>>> I am trying to design an RHQ/JON agent plugin to manage a software
>>> resource with the following characteristics:
>>>
>>> - there is the software itself (the installation);
>>> - there are a variable number of "bundles" of configuration information
>>> about 250KB in size each which need to be read from and written to the
>>> agent; and
>>> - there are "log" files which can 10-50MB in size each which need to be
>>> read from the agent.
>> We have the Event subsystem that can read log files and report them back
>> up to the server - one log message equals one RHQ event. However, this
>> is normally not used in an on-going basis for constant and continual
>> monitoring of log files. People usually just enable events on a resource
>> for a short time, to collect logs for a particular resource while doing
>> debugging or analysis. The backend event storage in RHQ is strictly in
>> the normal RHQ database and as such is not efficient for storing large,
>> ongoing log files (do not think of this as a "Splunk" type model). So
>> I'm not sure if the events subsystem is what you want, but at least its
>> something to look at.
> OK.  I looked at that and it basically scrapes error/debug messages from
> log files and turns them into events.  That would not be appropriate for
> the amount/kind of data we have.

>>> I think I am pretty clear on how to handle the software itself--just
>>> like any number of other agents.
>>>
>>> I am not sure how to handle the configuration bundles and the large log
>>> files.
>>> We might want to have the RHQ/JON server manage different versions of
>>> these configuration files and distribute them to multiple remote agents.
>> We used to have the concept of "Raw Configuration" (which was to support
>> raw configuration files) but time constraints among other things cause
>> us to stop that work (though some of that work still exists in the code,
>> its not fully baked, we plan on returning to that at some point). So,
>> this is a non-starter unless and until the raw config support is completed.
>>
>> We could look at the "ResourceFactoryFacet" which is how we do things
>> like deploy individual files to a resource (we use this, for example, to
>> deploy a WAR file to JBossAS). Its conceivable this can be used to ship
>> any file you want - this is the<content>   metadata tag in the descriptor.
>>
>> Finally, the new bundle subsystem can also be considered, however, its
>> purpose was mainly to support shipping out bulk distributions of entire
>> software installs (like a JBossAS distribution zip file or something
>> like that). But there is no reason why it couldn't be used to ship out
>> smaller bundles of content. I just committed into the master branch the
>> ability to push bundles to non-platform resources (see BZ 644328 at
>> https://bugzilla.redhat.com/show_bug.cgi?id=644328 ), which would
>> probably make this a bit easier and might make it be more acceptable to
>> use bundles for something like this.
>>
>> I'm just throwing things out there - you'd have to look at these
>> concepts and see what makes sense and if you want to use those to do
>> what you want.
> OK, it sounds like the ResourceFactoryFacet is the way to go, or the new
> bundle support.  I'll have to take a closer look.
>>> Is there some existing domain object that would handle the read/write
>>> aspect of the configuration bundles (zip files)?  Could the "package"
>>> concept be used for these?  Would we need to create a new domain object
>>> on the server side for these bundles?  If so, is there an example of
>>> this kind of thing?
>>>
>> Right - the "package" concept you are talking about is that
>> "ResourceFactoryFacet" /<content>   thing I mentioned above. That might
>> be useful here.
>>
>> Creating a new domain object or adding server-side EJBs cannot be done.
>> We aren't pluggable in that manner. That would require a fork/patch to
>> RHQ, and obviously mean you'd have to package and build RHQ yourself
>> with your changes.
>>
>> We are pluggable on the agent side using agent plugins, and we are
>> pluggable on the server side using server-plugins, but as for adding
>> domain objects and server-side business tier objects like EJBs, we
>> aren't easily pluggable there.
> To be honest, I didn't want to go this route.  ;-)
>>> For the log files, I see some mention of the SupportFacet.  Would this
>>> be appropriate for retrieving large log files?  Is there an example of this?
>>>
>> Yes, I believe (from just talking to you) you figured this out :) The
>> SupportFacet allows you to zip up any files you want. There is no
>> support for capturing snapshots in the GUI today, its only available
>> through the CLI. If you think this is a useful feature, we should
>> probably consider doing something in the GUI to have a better
>> integration here.
> Right, I did that through the CLI.  I expect the large log files would
> only be "pulled" from the agent in response to an alert, though it might
> be a regularly scheduled operation for tracking changes.  If I had no
> constraints, I would have that log file be stored by the server with the
> annotation of the alert or operation that caused it to be collected, and
> later someone could extract that from the server via CLI and analyze it
> in a separate tool, or launch the tool directly from the server GUI with
> the log file.  Or even more pie in the sky, an alert might trigger a
> change of configuration via the agent to collect diagnostic information
> for a set interval, and then resume normal settings and pull the
> diagnostic data back for storage.  I don't know if that is feasible (or
> desirable).
>>> We expect to access the configuration bundles and the log files using
>>> remote client operations because we have a separate GUI tool to
>>> build/edit the configuration bundles and to correlate and analyze the
>>> log files.  Is there an example of using a remote client to pull files
>>> from and push files to remote agents?
>> The architecture and design of RHQ is to specifically prohibit remote
>> clients from talking directly to agents. Servers talk to agents and if
>> remote clients want to talk to agents, they must do so indirectly
>> through the server over the remote server API.
>>
>> You could write a server-side plugin to act as a proxy to the agent and
>> do specific things, but the server plugin would still need to go through
>> the server-side EJB/SLSB API to talk to the agent.
>>
>> Your remote clients can push/pull content from the agent indirectly
>> using things like the ResourceFactoryFacet (the package stuff you were
>> referring to, though that is not really appropriate for log files and
>> config files and the like, that is more for things like content that
>> doesn't change - like versioned jar libraries and things like that) or
>> the SupportFacet or the Bundle subsystem.
>>
>> Of course, there is nothing stopping your custom agent plugin component
>> from exposing its own server-socket (for example) and listen for custom
>> remote client comm, but that just circumvents RHQ entirely. Still, that
>> is a possibility. Depends on what you want to do.
> Right, I didn't mean pushing/pulling to/from an agent directly, always
> through the server.
>> John Mazz
>> _______________________________________________
>> rhq-devel mailing list
>> rhq-devel at lists.fedorahosted.org
>> https://fedorahosted.org/mailman/listinfo/rhq-devel
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel


More information about the rhq-devel mailing list