Bug 1125343 - All content-related workflows broken on Agent
by Thomas Segismont
Hi,
I just hit the NPE on ContentServices problem again so I've reopen the
bug. The good news is now I know why it happens. Here's the issue.
All PC subsystems are initialized in
org.rhq.core.pc.PluginContainer#initialize
InventoryManager is initialized *before* the ContentManager
When InventoryManager is initialized, it activates already inventoried
resources recursively. Inventoried resources which have child types
creatable by content are given a
org.rhq.core.pluginapi.content.ContentContext instance. The problem is
the ContentContext holds a null reference to ContentManager, because at
this point the Content manager is not initialized, and not even created.
The consequences are the following:
* resources added to inventory *after* the PC started will be able to
use the content services, because when their resource context is
created, the content manager is already initialized
* resources *already* in inventory when the PC is started will not be
able to use the content services.
I'm working on a fix for 4.13.
Regards,
Thomas
9 years, 9 months
Introducing org.rhq.test.AssertUtils#timedAssertion
by Thomas Segismont
Hi everyone,
I was working on timing issues in our tests suite. Those issues often
happen when we have to wait for an async process completion:
* wait for alerts to be fired
* wait for a resource to be discovered
* ... etc
The common answer to that problem was often to raise the value of the
Thread#sleep parameter (sometimes from a few seconds to a few minutes),
thus making the build slower.
I wrote a change in the test-utils module. It's a new method in AssertUtils:
======
public static void timedAssertion(AssertUtils.BooleanCondition
booleanCondition,
String message,
long maxWait,
TimeUnit maxWaitUnit,
long step,
TimeUnit stepUnit)
throws InterruptedException
Asserts a condition evaluates to true before a timeout is reached.
Parameters:
booleanCondition - the condition to evaluare
message - the message to show if this assertion fails
maxWait - the maximum amount of time to wait before the condition
evaluates to true
maxWaitUnit - the time unit for maxWait
step - the amount of time to make the executing thread sleep between
condition evaluations
stepUnit - the time unit for step
Throws:
AssertionError - if the conditions does not evaluate to true before the
timeout is reached
InterruptedException - if the current thread is interrupted while waiting
======
I've just used it to fix an issue in the server itest suite. I hope it
will be helpful for everyone.
Regards,
Thomas
9 years, 9 months
RHQ 4.12 Released
by Jay Shaughnessy
It is my pleasure to announce the release of RHQ 4.12. Notable changes are:
* The UI has been updated to follow the PatternFly look and feel
(http://www.patternfly.org). This is not totally complete, please
give us your feedback!
* Several new Bundle features:
o Ant bundles now support the rhq:handover that can pass content
to the plugin for processing, such as native script execution.
o Ant bundles can now use properties defined using property or
rhq:property tags in their recipes.
o Ant Bundle rhq:url-archive and rhq:archive tasks now support the
destinationDir attribute for exploded archives.
* The AS7/EAP plugin now supports Versioned Deployments.
o For example, when deploying myapp-1.0.0.war RHQ now gives you
resource myapp.war with version 1.0.0. In this way history is
not lost when updating the version, the resource remains the
same and only the version changes.
* Plugin writers can now report MISSING the availability type in
their getAvailability() implementations.
o Used when the underlying resource is detected as physically
removed, this can result in automatic uninventory or ignore of
the resource.
* An important effort has been made to harden the Postgres plugin.
* The RHQ Storage Node component continues to mature.
* The RHQ Server now runs on top of EAP 6.3.alpha (previously it
ran on EAP 6.1.Alpha).
Release notes and zip download links are here:
http://docs.jboss.org/author/display/RHQ/Download+RHQ
If you want to try out RHQ without too much setup, a pre-created Docker
image will soon be available here:
http://index.docker.io/u/gkhachik/rhq-fedora.20/
We look forward to your feedback via:
* The RHQ Users mailing list:
http://lists.fedorahosted.org/mailman/listinfo/rhq-users
* The RHQ Developers mailing list:
http://lists.fedorahosted.org/mailman/listinfo/rhq-devel
* #rhq on irc.freenode.net
For their code contributions to this release, special thanks go to:
* Elias Ross
* Michael Burman
9 years, 9 months
storage cluster and maintenance mode
by John Sanda
When the server is in maintenance mode, it rejects all incoming requests from agents. When the storage cluster is down, we automatically put the server into maintenance mode. This came up while I was discussing with Michael Burman the work he is doing for adding support for changing the location of the rhq-data directory[1].
Changing the data directory will involve copying files and shutting down the storage node. It could potentially be a long running operation if there is a lot of data and/or slow disk. This can get a bit tricky when we have just one storage node. If it takes a while to move the data, there is a good chance that the server will go into maintenance mode. This could be problematic since the agent won’t be able to report the results of the operation back to the server. On the other hand, we cannot insert metrics while the node is down; so, maintenance mode kind of makes sense since we cannot process the measurement reports.
I am not sure about the best way to handle this. While the storage node is down, we could add the measurement reports to some form of a queue on the server. When the node comes back up, we would then drain the queue. Alternatively, we could change the behavior in the agent/server comm layer. As I understand it, if the server throws an exception when the agent sends a measurement report, the agent will immediately try to resend. We could add a delay in there so that the agent wouldn’t keep hammering the server.
I have not thought about this enough yet to arrive at an opinion. What do others think? And are there any other options that we ought to consider?
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1074636
9 years, 9 months
rhq-plugin.xml for tomcat-plugin
by jean-frederic clere
Hi,
I am trying to solve the following: I have readOnly field that can be null.
+++
<c:simple-property
name="configFile"
type="string"
readOnly="true"
description="The location of the context.xml
resource or file" />
+++
Gives an error when the field is empty
+++
<c:simple-property
name="configFile"
type="string"
readOnly="true"
required="false"
description="The location of the context.xml
resource or file" />
+++
That result in a very weird display when there is a value and allow to
entry something in the field was it is empty (and also to Unset it).
How can I get a readOnly field that can be empty?
Cheers
Jean-Frederic
PS: Re-posting after using wrong email...
9 years, 9 months
rhq-plugin.xml for tomcat-plugin
by jean-frederic clere
Hi,
I am trying to solved the following: I have readOnly field that can be null.
+++
<c:simple-property
name="configFile"
type="string"
readOnly="true"
description="The location of the context.xml
resource or file" />
+++
Gives an error when the field is empty
+++
<c:simple-property
name="configFile"
type="string"
readOnly="true"
required="false"
description="The location of the context.xml
resource or file" />
+++
That result in a very weird display when there is a value and allow to
entry something in the field was it is empty (and also to Unset it).
How can I get a readOnly field that can be empty?
Cheers
Jean-Frederic
9 years, 9 months
Is scheduled repair necessary with storage nodes?
by John Sanda
Repair is process in Cassandra in which data is made consistent across replicas. There are two kinds - read repair and anti-entropy repair. The former happens automatically in the background on queries. The latter is done via JMX. Although nodes can remain operational while anti-entropy repair, it is very resource intensive and can take a long time to run. It can easily be on the order of hours. The Cassandra docs recommend running regularly, scheduled anti-entropy within gc_grace_seconds, which is the time to wait before Cassandra garbage collections tombstones (i.e., deletion markers). The reason for running it within gc_grace_seconds is to ensure deletes get propagated and to prevent them from being undone. gc_grace_seconds is configured per keyspace and defaults to 10 days. We set gc_grace_seconds to 8 days, and we run anti-entropy repair weekly in a Quartz job named StorageClusterReadRepairJob.
After some investigation I am now of the opinion that we do not need to run a scheduled repair job. As long as replicas are up, data will be consistent between them. If we have cluster where the nodes never go down, then there is no need to run anti-entropy repair with respect to data consistency. Of course nodes do go down. Cassandra has another mechanism called hinted handoff that comes into play. When the target replica is down, the coordinator node (the one receiving the request), stores a hint of the mutation that is intended for that replica. When the replica comes back up, it will receive the hints, making it consistent with other replicas.
There is a maximum amount of time a node can be down and other nodes will store hints. The is defined by the max_hint_window_in_ms property in cassandra.yaml, and it defaults to 3 hours. If a node is down longer than that, then other nodes assume the down node is dead unless and until it comes back up. So if we do not run scheduled repair and if a node is down for more than max_hint_window_in_ms, then need to run a full repair on the node when it comes back up to account for any dropped hints.
As for deletes, I do not think we need to be concerned for a couple reasons. First, we are dealing with append-only data, where each column is only ever written once and never updated. Secondly, we write all data with a TTL. In the event some metric data was deleted on one replica, and still alive on the other, we know that it has the TTL set and will expire; therefore, we do not need to worry about deletes being undone.
These changes would make https://bugzilla.redhat.com/show_bug.cgi?id=1118099 irrelevant, but also require a new BZ to track the changes for running repair when a node is down longer than max_hint_window_in_ms. Unless anyone has objections/concerns, I will write up the new BZ and link 1118099 to it.
- John
9 years, 9 months
Configuration updates on agent
by Libor Zoubek
Hello,
Let me remind, how our configuration updates roughly work. From UI or
remote api comes request to update configuration for particular resource.
We write this into DB as last known configuration with status IN_PROGRESS
and send it to agent. Agent (as implemented in most plugins) updates all
configuration properties, it does not care about requested
changes - we do not provide such information to plugin (user does not
likely change all property values). Then, if
success is returned, server marks given configuration as gold and
successful.
Recently I was working on a bug, where AS7 reacts inappropriate when we
update particular attributes. While user changes only 1 config property
and does not expect any side effects and does not even know we're updating
all values. I've solved it with
https://github.com/rhq-project/rhq/commit/3d0571af3f465c6d47567483ebc0148...
which is a simplest AS7 only implementation of bellow ..
What would be the "correct" workflow for updating configuration?
Do we care about out of band config changes?
I am thinking of this:
1. new (full) config comes from UI/CLI
2. server sends full configuration to agent
3. agent reads live configuration
4. agent creates a configuration diff
5. agent sends this diff to plugin to apply changes only
by configuration diff I mean some object or maybe visitor with callbacks
that only marks configuration property changes
or similar could be that steps 3-5 does plugin itself and just provide
generic methods for creating configuration diff.
I know this workflow introduces additional read of configuration prior
write, but I don't see it as an issue, because configuration updates are
more or less one-off operations.
Thanks
--
Libor Zoubek
9 years, 9 months