Apache resource upgrade integration tests

Lukas Krejci lkrejci at redhat.com
Tue Jul 12 11:37:12 UTC 2011


On Monday, July 11, 2011 20:51:05 Charles Crouch wrote:
... snip ...
> Can you elaborate on what state the system will be in if an upgrade "fails"
> and also the expected type of next steps are a user should take to get
> themselves into a consistent state, assuming state after upgrade "failure"
> is inconsistent.

Upgrade can fail for two of reasons:

1) the upgrade method of the discovery component of certain resource type 
throws an exception - i.e. fails to upgrade a resource.
2) 2 or more sibling resources would have the same resource key after the 
upgrade.

In either case, the following will happen:

1) All sibling resources of the failed ones are permanently stopped until the 
upgrade error "clears" (more on that below), i.e. they all appear red in the 
UI.

2) The resources that failed upgrade are assigned a resource error of type 
"UPGRADE" with a detailed description of the upgrade failure (i.e. the 
exception message in case of the upgrade method failure, or the list of the 
conflicting resources in case of resource key duplicities). The error is 
visible in the detail page of the UI after clicking on the "warning triangle" 
icon next to the availability icon.

The user is then expected to do the following:

1) Upgrade method failure:
a) more than 1 vhost could match the resource key:
   i) Upgrading from RHQ 1.3.x:
      The triggering scenario for this is quite convoluted:
      The apache config would have to change order of the vhosts in the config
      files so that the SNMP WWW Service index would no longer correspond
      (thus making the vhost resource basically invalid because it would
      monitor a different vhost than it appears to represent). In addition to
      that, the vhost in the apache config that corresponds to the SNMP index
      would have to have the same ServerName as one or more other vhost in the
      apache config (this is legal, but confuses RHQ 1.3.x plugin which isn't
      able to distinguish between such vhosts).
      The only universal advice here is just to uninventory the vhosts and let
      RHQ discover them anew because the monitoring was broken anyway.
   ii) Upgrading from RHQ 3.0.2: 
       there is a mix up in the SNMP WWW service indices. The user 
       is advised to consult the output of the "Detect SNMP WWW Service
       indexes" operation on the apache server resource to determine 
       what indexes the vhosts should have and edit the plugin configs of 
       the vhosts to assign the correct SNMP WWW service index value there.
   iii) Upgrading from RHQ 3.0.0 or RHQ 3.0.1:
        there is not much for the user to do here, because the old resource
        key is ambiguous and there's no way to provide more info to 
        the plugin. The only thing that the user can do is to uninventory the
        failed vhost resources.
b) no vhost could match the resource key:
This can happen for a main vhost which has a resource key that doesn't 
correspond to the URL property in the connection properties of Apache server
resource and SNMP is either not configured or RHQ 3.0.[01] code fails to
determine the SNMP index from the SNMP report. In such case though there
should be another main vhost discovered that corresponds to the URL property.
The advice here is to uninventory the main vhost that has the resource key 
different to the URL property.

2) Multiple vhosts upgrade to the same resource key:
This could happen in RHQ 3.0.[01] if Apache server was first discovered 
without SNMP support and SNMP would be later configured. In such case, 2 vhost 
resources would appear for each of the vhosts, one with RHQ4-like resource key 
and one with RHQ1.3-like resource key.
The user is free to pick which one of the duplicate vhosts she wants to keep 
(possibly the one on which alerts are defined) and uninventory the other.


Once the upgrade error is resolved using one of the above methods, the plugin 
container of the agent has to be restarted to pick up those changes and 
successfully upgrade the resources.



More information about the rhq-devel mailing list