resource type ignoring
by John Mazzitelli
[this is more for charles - but I want to put this on the devel list to have a transparent and public thread]
I would like to start a thread dealing with BZ 535289 (aka RHQ-1) - this was a use-case being discussed:
> https://bugzilla.redhat.com/show_bug.cgi?id=rhq-1 specifically:
>
> Suppose I have machines with multiple agents, each potentially
> having ~40k child service resources of a particular type in a given agent
> inventory.
>
> Suppose I do not care about all of those resources, but I do care about some of them.
> How can we support monitoring certain resources while not all 40k.
This is where RHQ-1 comes in - the ability to "disable" resource types (or, said another way, ignore a bunch of resources of a given type).
There are three things here.
1) Be able to disable all new resources for a specific resource type. So, if I go to a (as yet non-existent) Administration page, I click some buttons to say "disable type 'foo'", then anytime the agent sends a inventory report with resources of that 'foo' type, we ignore them. They never get into the discovery queue and never get a row in RHQ_RESOURCE table. If they are already in the discovery queue, you already have the option to "IGNORE" them, so that's already a feature and has been for a while. But what if their parent is in the queue? (the parent is not an ignored type). If we commit the parent, the agent will send up the child services of type "foo" and we simply ignore them - we don't commit those ignored resources and we throw them away.
This is the "all of them" approach - we ignore all resources of a given type.
2) Be able to ignore a select number of resources already in inventory. We could do this from the inventory pages - I select some resources, click a (as yet non-existent) "Ignore" button, and they get ignored - hidden from view - and the agent will somehow need to disable or ignore them as well. This sounds just like uninventory - but the problem is, the agent will simply rediscover them during the next discovery run. We need to somehow tell the agent "uninventory these resources and don't discover them again". How do we indicate this inside the agent? I don't know. We probably need to keep the resource in the agent's inventory with a status of IGNORED. But if we have 40,000 resources, and we want to IGNORE all of them (or at least most of them), the agent's memory utilization could be large for holding resources we don't care about. I don't know a way around this - we may have to just suck it up and use the memory.
3) Be able to not ignore a small amount of resources, but ignore all the rest. This is very similar to #2 above, but this goes to the fact that I might have an agent discover 40,000 resources, but I only care about, say 100 of them. How do we do this?
This #3 is the most difficult one. How do we ask the user to select 39,900 resources and press the ignore button? Not realistic. Not only that, but in the case where those 40k are child services, today we never get a chance to accept or reject them from the discovery queue - once the parent is committed, only then does the server even get to know what's underneath it (the agent will only discovery child services after the parent is committed).
I think this may involve a big change - in the discovery queue, we may have to support a new option when pressing the Import button - somehow we need to support an option "do not committed child services". This would then commit the server but then somehow show the child services in the discovery queue, at which time you can pick which ones you want to commit to inventory (thus, you only pick 100, and you keep the other 39,900 in the queue in NEW state - I suppose you could opt to then IGNORE them? Many problems abound with this. The UI tree in the queue view doesn't support N layers deep - only the top platform and the immediate child servers are shown. Also, how do we show 40,000 (or something that large) rows in the tree? Do we hit performance problems in the UI?
This is a huge change - from the discovery queue UI down to the server then to the agent, changes would need to be made to support this.
So, #1 and #2 should be doable (though difficult to fully implement). #3 is very difficult and would require some design changes and some deep thinking as to how this would effect all tiers of RHQ. At this point in time, I'm thinking we just don't attempt #3.
11 years, 1 month
Re: Evolution of hudson plugin
by Jérémie
2013/3/12 Thomas Segismont tsegismo at redhat.com :
>Le 09/03/2013 18:16, Jérémie a écrit :
>> 2013/3/6 Thomas Segismont tsegismo at redhat.com :
>>> Le 06/03/2013 23:50, Jérémie a écrit :
>>>> Hi rhq team,
>>>>
>>>> I want to create an alert if a hudson job has not been built for more
>>>> than two days. It seems to me that isn't possible on trait.
>>>>
>>>> What do you think of adding two metrics on the hudson plugin giving
>>>> the time elapsed since the last build, and since the last successful
>>>> build. lastSuccessfulBuildElapsedTime and lastBuildElapsedTime.
>>>>
>>>> Attached is a patch that seems appropriate for my use case.
>>>> Your feedback are welcome. I can create a jira for it.
>>>>
>>>> Regards,
>>>> Jeremie.
>>>>
>>>> _______________________________________________
>>>> rhq-devel mailing list
>>>> rhq-devel at lists.fedorahosted.org
>>>> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
>>>>
>>>
>>> Hi Jérémie,
>>>
>>> I'm glad to see a contribution coming from France :) (well, seems to
>>> according to your surname orthography).
>>>
>>> Can you please log a new Bugzilla entry, prefix its name with 'RFE' and
>>> attach your patch there? That would be a good start for it to be merged.
>>>
>>> Bugzilla URL: https://goo.gl/xlmvJ
>>>
>>> Thanks and regards,
>>> Thomas
>>
>> Hi Thomas,
>>
>> Thank you for your advice.
>> I created a jira here https://bugzilla.redhat.com/show_bug.cgi?id=919248
>> Your comments are welcomed.
>>
>> Regards,
>> Jérémie.
>> _______________________________________________
>> rhq-devel mailing list
>> rhq-devel at lists.fedorahosted.org
>> https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
>>
>
>Hi Jérémie,
>
>Your patch looks good, in order to get it incorporated you must create a
>Fedora Project account (if you don't have one already) and sign the
>contributor agreement?
>
>You will find instructions here:
>https://docs.jboss.org/author/display/RHQ/Contributions#Contributions-The...
>
>Once this is done, can you give me your Fedora account id?
>
>Cheers,
>Thomas
>
Hi Thomas
Thank you for taking the time to look at my patch.
My Fedora account name is : jeremie
Regards,
Jérémie
11 years, 1 month
remoting dependency upgraded - rebuild may be necessary
by John Mazzitelli
I just completed https://bugzilla.redhat.com/show_bug.cgi?id=921704
This is the upgrading of our JBoss/Remoting 2 dependency - we were on 2.2 but we are now on 2.5.4.SP4. The purpose of this is to provide better IPv6 support (plus, I think it helps further prepare us to move to Remoting 3 which we *may* have to do in the future).
I think this change is going to require you to do a rebuild of your container. For example, a new remoting servlet needed to be deployed in the EAR (we have a new Remoting WAR module that takes the place of a patched version of the old Remoting WAR that we were using).
So, if you see any odd behavior with agent<->server communications, first thing to do is do a rebuild of both server and agent.
I ran all unit tests and integration test and all pass. I tested sslservlet, servlet, sslsocket and socket protocols on server and agent and things work.
I also ran the CLI and confirmed that that can talk to the server over port 7080 and 7443 (I didn't see a place where you can tell CLI to go over an SSL transport, I assumed if I told it to go over port 7443, it knew to go over the secure transport, because it works. But what happens if a customer changes the 7443 port to something else? how does the CLI know it should go over a secure port? i don't know).
So, I touched all the normal places and things all look to be working.
If you see any problems wrt low level communications even after a full rebuild, submit a BZ immediately.
That is all.
John Mazz
11 years, 1 month
Alert Maintenance & Permissions
by Mark Addy
Hi,
I've been looking into an issue reported by our users around the
acknowledgement and deletion of alerts.
From the main dashboard, or any view not associated to a group or
resource the only users capable of acknowledging or deleting alerts are
those with global permissions. The effect is that users with specific
role permissions to manage alerts against a group of resource must drill
into the heirarchy before they can ack or delete those alerts. We'd
like to be able to let them perform this action from any of the recent
alert windows.
After looking into the code I the fix seems straight forward.
A change the "Delete" and "Acknowledge" alert buttons in the
AlertHistoryView to use a "ResourceAuthorizedTableAction" rather than
"AbstractTableAction". This will generate an additional call to the
backend to check permissions when an alert row is selected and activate
the button if permission exists. This could be optimized by
pre-generating the users permissions as part of the Alert Table as
suggested in the comments in "ResourceAuthorizedTableAction" - I could
then override the "isEnabled" method to use these rather than calling
the back end. This would require creation of some kind of
"AlertWithPermissions" Composite to wrap the "Alert" and include the
users permissions or a manageable flag?
I'd like to make the changes and contribute them back - we are currently
running 4.4.0 but the change also applies to the current master branch.
Would you be happy with the changes suggested above or perhaps there is
a better way to implement this?
If its Ok what is the best way for me to submit changes and against
which branch should I make them?
Thanks
Mark
11 years, 1 month
GitHub migration again -- The Plan
by Heiko W.Rupp
Hi,
I've updated the wiki page including "The Plan".
I would like to finish this Plan within the coming 7 days so that
we will be able to switch over soon.
https://docs.jboss.org/author/display/RHQ/GitHub+migration#GitHubmigratio...
Please comment. If you think it is ok, also say so please.
Heiko
--
Reg. Adresse: Red Hat GmbH, Technopark II, Haus C,
Werner-von-Siemens-Ring 14, D-85630 Grasbrunn
Handelsregister: Amtsgericht München HRB 153243
Geschaeftsführer: Mark Hegarty, Charlie Peters, Michael Cunningham, Charles Cachera
11 years, 1 month
RHQ 'plugin store'
by Ken Rumer
This is a request for enhancement in the agent plugin area of RHQ. There
are several products doing a great job getting community involvement
because they basically allow anyone to submit software, and make it very
easy. I like the interface provided by Jenkins, and other examples would
be apple or google stores. Basically, I would like to be able to upload *any
*plugin, good or bad, and the users would be able to rate the plugin and
determine if it has value. For example, you may have 3 different versions
of an IIS plugin, because some users may want a complete monitoring of IIS,
while some may just want to see up/down.
Note: I don't like having to merge source with RHQ's code, I may not follow
RHQ standards, or maybe my code breaks something in core??? I'm not a
developer, I'm a Middleware admin, if I were RHQ, I would want Ken Rumer's
code as far separated from RHQ as possible. :)
11 years, 1 month
Cassandra - Data Migration
by Stefan Negrea
Hello Everybody,
After a few weeks of slowly collecting data and then refining and optimizing the import code I have some good numbers to share.
For this exercise I created a tool [1] that generates random data based on the estimates I've previously done [2]. I used two models 1)everything local (Cassandra, migration tool, postgres database) and 2)remote postgres database but local Cassandra and migration tool. With the remote postgres, I had a somewhat slow network connection to the server but the hardware was much better than I had locally.
The migration tools works in two modes:
1) migrated everything, then delete everything at once
2) migrated a little bit of data, then delete what got migrated and repeat
Here is what I can report so far:
1) Test data creation is resource intensive but the real bottleneck is the SQL insert process. The time required to insert test data for 1 agent slowly increases over time. When I used the remote postgres server, the standard deviation to insert data for 1 agent was low. So the insert process slows down less with better hardware (expected). Some telemetry for this can be found at [3].
2) The observations from test data generation are very useful in case we decide to make the gradual data migration default (the second migration process). Deleting migrated data can consume a lot of time for the initial batches but decrease over time as there will less data left in the database.
3) Average time to migrate data for 10 agents is between 20 to 27 minutes with everything local and data deletion at the end. The CPU utilization is high (90% or above) because Cassandra, the migration tool and postgres are all crunching numbers.
4) Average time to migrate 10 agents is between 35-45 minutes with a remote SQL and data deletion at the end. I attribute this difference on the slow connection between my box and the remote postgres server. The CPU utilization was lower (50-60%) which means the bottleneck was the network.
5) The time required for migration does not scale linearly. Migrating 20 agents will take slightly longer than two independent 10 agent migrations. Because of the indexing on Cassandra side, more data added means more indexing workload.
6) The average time required to migrate 40 agents is between 1.5hours and 2 hours with everything local. Still running tests with the remote postgres server for 40 agents.
7) I am still gathering telemetry for migrating 125 agents but some of the early runs were north of 7.5 hours. The code has received many optimizations so that time is be lower now but not half.
8) With everything local, I could see about 1 to 1.2 million raw metrics records per minute migrated to Cassandra.
9) The amount of memory required by the migration tool is around 700 MB. That is mainly because the migration tool processes a batch of 30K records from SQL at once.
10) If I continue to optimize the import tool I do not expect to reduce the total amount of time by more than 25% because of the large of amount data/processing required. Data flows from postgres, to the migration tool, and into Cassandra. At each step there is substantial processing; for example, the migration tool needs to convert the data from the old model into the new model.
So based on these numbers, the migration would take a few hours for large deployments. If we choose to do the migration with the RHQ server down then there will be no RHQ UI capability for a few hours; metrics will still be collected by the agents but not reported to the server for a few hours. Is that acceptable?
On the other hand (as discussed in previous emails), if migration is happening with the server up then aggregates will be skewed until the full data is migrated. Also, if the RHQ server is running, the migration process will need a little slowdown so it does not gobble all the server CPU. Is this better?
Any thoughts?
Thank you,
Stefan Negrea
Software Engineer
[1] http://git.fedorahosted.org/cgit/rhq/rhq.git/tree/modules/enterprise/serv...
[2] https://docs.jboss.org/author/display/RHQ/Metrics+Data+Migration+-+Design
[3] https://docs.jboss.org/author/download/attachments/56885438/GenerationTel...
11 years, 1 month
RHQ/Cassandra architecture
by John Sanda
There has been a lot of discussion recently around how RHQ and Cassandra should co-exist. The decision has been made to proceed with the following,
* Installing RHQ server will also install Cassandra and agent by default.
* Advanced options will be available to install RHQ server by itself or just Cassandra and agent.
* Managing/monitoring Cassandra will be done via the agent.
* Starting/stopping the server will by default start/stop everything that is installed on the machine.
Rather than changing existing scripts like rhq-server.sh to provide this augmented functionality, what do you think about an rhq-uber script (e.g., rhq.sh) that handles these changes? If I execute,
$ rhq.sh start
The the script will start Cassandra, the server, and the agent, assuming a default installation. The nice thing about this is that if the user only wants to start/stop the server, he can instead directly use the already familiar rhq-server.sh script; however the uber script would also provide that capability as well. Maybe something like,
$ rhq.sh start --server-only
The new script would provide for centralized command and control of everything installed on the box.
- John
11 years, 1 month
new graphs
by John Mazzitelli
I have two comments on the new graphs.
1) The fonts in the X and Y axes are fat and fuzzy. Is there anyway we can use better fonts? Its hard to read some of the labels because the letters are so fat and fuzzy they bleed into one other.
See attached font.png as an example. You have to look really hard to read what that Y-axis label is saying. And even though the X-Axis times are readable, they aren't crisp and easy to read.
2) I think the lines that connect the bars should definitely not go down to zero for the "grey" bars - that is, for the times when no collection was made. The lines infer the value of the metric is zero when that is not true - we have no idea what the value is for the metric at that time. To show the line connecting at 0 is misleading. When you look at the graph, they all look saw toothed - as if the metric values are some number, then 0, then some number, then 0 which is not necessarily the case (and probably isn't the case). We should only connect the bars where we have actual values.
See sawtooth.png for what I'm talking about.
11 years, 1 month
bundle / content api redesign
by John Mazzitelli
(This is more for Lukas, but it seems appropriate to discuss over the rhq-devel ML)
Lukas,
Jay and I went through the Bundles vs. Content design page,
https://docs.jboss.org/author/display/RHQ/Bundles+vs.+Content+-+On+Its+Head
In general the ideas sound good.
However, there are two things I'm questioning:
1) this is a major re-design/re-implementation - I'm not sure if we need to be concerned with backward compatibility with the older content API (i.e. I think we should release this in RHQ 5 timeframe, the new major release should indicate it contains some backward incompatible changes with RHQ 4).
2) (this is the bigger concern), it seems like you deferred to maintaining the Content/Package API and deprecated the Bundle API. In my opinion, we should do this in reverse. We should maintain the Bundle API (merging in the Content/Package API where applicable) and deprecate the Content/Package API.
Bundles has been a successful feature - more people are using it and like it (and they are using it successfully for things we didn't anticipate) compared to the Content/Package API (few people use it, and fewer like it :-)
The Content/Package classes are what needs to be deprecated, not the Bundle classes. I would recommend that where you say things like:
Bundle
@Deprecated NON_ENTITY computed from Package
...
BundleVersion
@Deprecated NON_ENTITY computed from PackageVersion
should instead be flipped like:
Package
@Deprecated NON_ENTITY computed from Bundle
...
PackageVersion
@Deprecated NON_ENTITY computed from BundleVersion
and things like:
PackageVersionDeployment NEW CLASS
exact copy of BundleDeployment, which become @Deprecated NON_ENTITY
...
PackageDestination NEW CLASS
copy of BundleDestination, which becomes @Deprecated NON_ENTITY
should instead just use the Bundle classes (BundleDeployment, BundleDestination) and just add data fields to the entities as necessary.
We can discuss further as to why Content/Package API is being retained and Bundle API is being deprecated, but those are my immediately thoughts.
11 years, 1 month