Resource upgrade once more

Lukas Krejci lkrejci at redhat.com
Thu Sep 16 10:19:59 UTC 2010


On Wednesday, September 15, 2010 17:48:47 Joseph Marques wrote:
>   On 09/14/2010 12:04 PM, Lukas Krejci wrote:
> > Hi team,
> > 
> > there is an unfortunate one last issue with the resource upgrade
> > functionality that I'd like to get a common agreement on.
> > 
> > But first let me recap the state of resource upgrade functionality as we
> > decided to implement it.
> > 
> > 1) It will only be possible to upgrade resource key, name and
> > description, with the latter two being disabled on the server side (it'd
> > be easy to add a GUI switch to switch it on if we see the demand in the
> > community).
> > 
> > 2) An upgrade error isn't considered fatal, it is assumed that plugin is
> > either programmed to handle multiple versions of resource data or that
> > the failure to upgrade isn't detrimental to the plugin functionality
> > (too much).
> > 
> > 3) An upgrade error is going to be merely reported using the resource
> > errors facility on the server side similarly to how
> > InvalidPluginConfigurationExceptions are/were reported (some kind of
> > warning icon and explanatory text).
> 
> Keep in mind if you piggy-back on the resource error conceobject will
> only contain pt for upgrade errors, that this state can be
> lost/overridden by future plugin configuration update - whether it
> originated from the resource or the group.  If the future plugin
> configuration update is successful, the previous message about the
> upgrade failure is lost; but even if the plugin config update fails, the
> error (since it only represents the most recent issue) will override the
> information about the upgrade failure.  So if the plugin configuration
> update is kicked off from the group-level, the user/operator might not
> even see the error at all.
> 

I'm not completely piggybacking on the resource errors but rather enhancing 
it. AFAIU, the logic is to keep at most 1 error per error type for a resource. 
Therefore I added a new error type and followed the same logic. In the 
resource-upgrade branch, you'll see that ResourceErrorType enum contains a new 
value called "UPGRADE" (I think) and the 
ResourceManagerBean.addResourceError() method is updated to handle the new 
type of error in the same way as the plugin config and avail check errors.

> > So now let me explain the issue. The original intention of the whole
> > resource upgrade effort was to be able to transform an "old" resource
> > with a "new" resource (transform only means to change its resource key
> > for now).
> > 
> > Now that we have that functionality, let's examine what happens if the
> > upgrade fails for some reason (a logic error in the plugin code being
> > hopefully the only possibility here).
> > 
> > 1) There is the old resource in the inventory and a new plugin has been
> > deployed to the server.
> > 
> > 2) Agent downloads the new plugin and reinitializes the plugin container.
> > 
> > 3) The resource upgrade kicks in, upgrade of the old resource fails, an
> > error is reported on that resource on the server side.
> > 
> > 4) A discovery using the new plugin code is executed on the agent.
> > 
> > 5) A new resource is discovered that should have really been the
> > transformed old resource if the upgrade succeeded.
> > 
> > 6) We end up with two logically equal resources in the inventory.
> 
> So logically equivalent, but with differing resource keys, right?  One
> has the resource key from the old plugin, and one has the resource key
> from the new plugin?
>

Exactly. 

> > The only way how to prevent this from happening that I can see is to
> > disallow the discovery for that certain resource type under the old
> > resource's parent for the lifetime of plugin container (that is until
> > the next upgrade attempt is made with the next start of plugin
> > container).
> 
> First, what are the expected failure scenarios here?  Are we presuming
> that we can't change the value of a string (in this case representing
> the resource key) to some new value?
>
 
I think the main goal for us here should be to prevent the "two logically 
equivalent resources" problem at all cost. In fact, it is THE issue we are 
trying to solve with the whole resource upgrade effort. By not doing anything 
in the case of upgrade failure, we are basically only making the problem much 
more rare than it would be without resource upgrade (without resource upgrade, 
this would happen any time a plugin changed a resource key format).

Resource key is not a user-friendly value and we probably should not ask the 
user to mess with it in any case. We could for example detect the upgrade 
failure and have some special kind of workflow asking the user "Hey, this 
resource failed to upgrade. Since then these new resources were discovered. 
Which one of them should replace the failed resource if any?". But that is 
quite complicated IMHO because we'd have to change the way resources are added 
to the inventory - we'd have to disallow auto-import of services in this case 
for example.

> Although the above suggestion seems like a possible catch-all routine,
> I'd like to understand the precise failure conditions we anticipate
> here, so we can investigate whether there are more reasonable ways of
> handling those specific types of failures.
>

There are 2 error conditions during the upgrade:

1) The ResourceUpgradeFacet fails to upgrade a resource (this is the problem 
we're discussing here).
2) Comms error while sending the upgrade requests to the server.

The second error condition is maybe even worse than the first one, because in 
that case no resources on the agent are upgraded (to be kept in sync with the 
server) but new plugin code would be in place and therefore discover the "new 
resources" for any resource type.

> > But is that acceptable? This way we would basically "conserve" parts of
> > the inventory in the old state until a new version of the plugin with
> > the fixed upgrade code would be deployed, which can mean a substantial
> > amount of time with possibly non-functional resources.
> > 
> > Another way how to fix this would of course be the plugin versioning and
> > multiple versions of the same plugin being deployed on the agent serving
> > different resources, but that's a very complicated issue that we agreed
> > to avoid at all cost.
> 
> Is it possible to ignore the new version of plugin A if there are
> upgrade errors...and just use the existing (a.k.a "old") version of
> plugin A, which is presumed to have been working?  This, understandably,
> becomes more complicated because of dependencies between plugins, which
> may require all plugins in the connected dependency graph to be rolled
> back as well...and is that even possible if some resources have already
> been upgraded?
> 

The rollback was one of the scenarios we discussed and decided not to pursue. 
But in the light of the above problems, it might be the best idea actually :)

No resources are upgraded on the agent or server before all the upgrade 
requests are collected from the plugins. Therefore, if we detect a failure for 
certain resource, we could finish the upgrade removing the resources of the 
affected resource types from the set of the upgrade requests (determining 
these is the hard part) and then report to the "owner" of the PC (i.e. the 
agent) what happened and stop the plugin container right during its startup. 
The agent then would replace the affected plugins with their old versions and 
start the PC again.

Note that this approach would gracefully handle the comms error scenario as 
well.

We'd also have to come up with some new way of communicating this kind of 
problem to the user because we could no longer use the resource errors 
facility for this.

> > What are your thoughts?
> > 
> > Thanks,
> > 
> > Lukas
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/rhq-devel
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel


More information about the rhq-devel mailing list