inventory connection changes and/or different use cases with auto-discovery / adding resources manually / plugin updates / resource key purpose

Lukas Krejci lkrejci at redhat.com
Mon Jan 3 14:21:28 UTC 2011


On Monday, January 03, 2011 15:05:56 Andreas Dietrich wrote:
>   Hi Lukas,
> 
> thanks for explaining the connections to me :-)
> I think your suggestion to differentiate between the physical resource
> and the view reflected in two different resource types makes much more
> sense. Thanks very much for this one.
> 
> The only problem left to me now is that I don't want to update our quite
> stable RHQ_3_0_0 environment right now, but would like to clean up the
> old resource keys we have (we actually don't have those duplicates/views
> I mentioned below right now).
> Thus ... what do I have to do to clean up my resource keys? Would the
> following be enough? (I guess I have to update it on the agent-side as
> well somehow):
> 
> 1) shutdown RHQ server
> 2) execute on DB: update rhq_resource set resource_key=name /* some hash
> = filePath */ where ... /* plugin='Logreader' */ ...
> 3) ? // update agent resource keys accordingly
>
> I hope 3) would not be too complicated.
> 

3) is actually quite trivial :) Just shutdown the agent and start it again 
with the --purgedata option. This will delete the agent side inventory and 
will redownload the inventory as it exists on the server.

So in your scenario (quite complex I must say), I'd do the following:

1) shutdown the server and all the affected agents.
2) sanitize your resource keys in whatever way you see fit in the server's 
database.
3) deploy your new plugin that will be compatible with those new resource keys 
to the server.
4) start the server
5) start the agents with --purgedata

The agents will start with clean plate and will download the new plugin and 
inventory from the server.

> Thanks a lot
> Andreas :-)
> 
> On 01/03/2011 02:47 PM, Lukas Krejci wrote:
> > Hi Andreas,
> > 
> > If I understand your usecase correctly, you basically want to have
> > different views on the log files.
> > 
> > The difficulties you're describing are real but I don't think we would
> > solve them by adding some more logic and asking plugins questions like:
> > "you just discovered this resource. Given the current inventory, is it
> > really valid?"
> > 
> > This would potentially complicate matters very much and I think it would
> > conceptually break our approach of the resource key being a unique
> > identifier of a resource.
> > 
> > Your log file views don't exist as standalone things outside of the RHQ
> > inventory. I think this is the fundamental cause of your difficulties
> > with RHQ way of doing things but at the same time, I think the resource
> > upgrade functionality can still help you. Basically, if your RK
> > calculation changes, the only person that knows about how to handle the
> > changes is you as the plugin dev.
> > 
> > I also think that by rearranging your resource type hierarchy you could
> > make your life easier.
> > 
> > What about this:
> > 
> > LogFile (RK = path)
> > -- View (RK = a random string, only manual add supported, with a
> > "default" view maybe autodiscovered, the config attributes would be set
> > in the plugin config)
> > 
> > With this approach, you get rid of the problem of LP vs. LP+. Because
> > View's RK is just a random string, you also get rid of the ARM problem.
> > It basically leaves the responsibility of "duplicate detection" on the
> > user (which I think is fair given the fact that views are only manually
> > added).
> > 
> > Even if you want the duplicate detection to work, you could base the
> > View's RK on the concatenation of your config attributes and have the
> > resource upgrade facet to "move" the resource from one key to another.
> > Because RK is the agent-side ID of a resource there are checks in place
> > that prevent two resources with the same RK (under the same parent) to
> > exist in the inventory.
> > 
> > Other answers are inlined below.
> > 
> > Thanks for the input, this is turning into a very interesting debate :)
> > 
> > Lukas
> > 
> > On Monday, January 03, 2011 13:36:28 Andreas Dietrich wrote:
> >>    Hi,
> >> 
> >> there seems to be another related and important problem to the whole
> >> issue with resource keys, auto discovery, adding resources manually and
> >> updating the plugin (configuration):
> >> 
> >> ===============================================================
> >> To explain it I'd like to explain it with help of our own case:
> >> 
> >> - L) we have a Logreader plugin in place with various config attributes
> >> (logfilePath, errorPattern, warnPattern, excludePattern,
> >> skipInitialLines, ...)
> >> 
> >> - what we want from each side regarding the resource keys:
> >> 
> >> -- AD) auto discovery: if we discovered a log file we do not want to
> >> discover it again with another resource key
> >> 
> >>       =>  LP (log file path only approach): logfilePath should be
> >> 
> >> sufficient as resource key
> >> 
> >> -- ARM) adding resources manually/Logreaders: we would like to be able
> >> to add different Logreaders for the same log file
> >> 
> >>       =>  LP+ (log file path + other attributes approach): logfilePath +
> >> 
> >> all or part of the other config attributes (see L) above) should make up
> >> the (initial) resource key
> >> 
> >> - U) if we update our Logreader plugin and have added some new attribute
> >> that has to go into the resource key calculation, we would basically
> >> auto-discover every existing Logreader when using LP+ because the
> >> resource key calculation has changed
> >> 
> >>       =>  IR (ignore auto discovered resources on plugin attribute
> >>       update
> >> 
> >> approach): we would have to ignore all the newly discovered resources
> >> (which is not always simple to do because one has to check if there is
> >> no Logreader that is really newly discovered and a duplicate of an
> >> existing one)
> >> 
> >>       =>  SR (import the newly discovered resources and uninventory the
> >> 
> >> old ones approach): this is something we don't like as explained in
> >> detail here:
> >> 
> >> http://metlos.wordpress.com/2010/10/06/upgrading-resources-in-rhq/
> >> 
> >> =>  since LP and LP+ are valid requirements, but go against each other
> >> with the current RHQ implementation, I think they should not be based on
> >> the same "resource key" definition
> > 
> > As I tried to explain above, I don't think this is completely true. In
> > fact, the usecase you're describing is fairly a non-standard one,
> > because you're trying to create "logical" resources that you let your
> > users define as opposed to the conventional approach of RHQ where each
> > resource corresponds to "physical" thing installed on the machine on its
> > own.
> > 
> >> ===============================================================
> >> S) (solution to the LP<->  LP+ / AD<->  ARM<->  U conflict)
> >> 
> >>   From my point of view we basically only need the "resource key" for
> >> 
> >> auto-discovery-identity-management (LP approach above).
> >> 
> >> (=>  We could think of it as "auto discovery resource key" or "if auto
> >> discovery finds a new resource, have a look if we should treat it as new
> >> or existing =>  thus compare the resource keys")
> > 
> > If autodiscovery generates the same resource key, the resource is not
> > considered new, but rather discarded, because resource with such RK
> > already exists.
> > 
> >> If we still need it for ARM: If we wan't to add resources manually there
> >> could be a hint "Another resource with the same logfilePath already
> >> exists. Are you sure to continue adding this resource (configuration)?".
> > 
> > I think this already is in place since RK is the agent-side ID of a
> > resource. Two resources with the same RK (under given parent) can't
> > therefore be allowed anywhere in the inventory. This is btw. also
> > checked for during the resource upgrade. If you upgrade a resource and
> > the new resource key is equal to the RK of one of the siblings, the
> > resource upgrade fails for that resource and the RK is not changed.
> > 
> >> S.1) The resource key should now be the same as the existing one, e.g.
> >> "/mypath/logfile.log" and thus it would not be a unique sibling anymore.
> >> Does this matter so much?
> > 
> > Yes, it does, because RK is the agent-side ID of a resource.
> > 
> >> S.1.U) Additionally this would solve the problems related to U since the
> >> logfilePath most likely stays the same.
> >> 
> >> 
> >> ===============================================================
> >> Some comments to other thoughts and their implications I had
> >> implementing something else than S.1:
> >> 
> >> S.2) The resource gets a new id and a resource key appended by e.g. some
> >> smallest positive integer>= 1 so the resource key does not exist yet,
> >> e.g. "/mypath/logfile.log (3)" if "/mypath/logfile.log (2)" already
> >> exists. (One should know this concept from MS Windows/Office behaviour
> >> ;-))
> > 
> > If this was the approach for all the resource types, you'd duplicate your
> > inventory on each discovery run, e.g. each 15 minutes ;)
> > 
> >> =>  S.2.ADC) auto discovery conflict: if we would uninventory the first
> >> discovered resource "/mypath/logfile.log" and have "/mypath/logfile.log
> >> (3)" left we would again discover a new resource with key
> >> "/mypath/logfile.log" without further "investigation" ala "resource_key
> >> like '/mypath/logfile.log%'" (S.2.ADE - auto discovery duplicate
> >> resource key exists algorithm).
> >> =>  S.2.U) U would be solved with S.2.ADE
> >> =>  why so much effort if S.1 is sufficient
> >> 
> >> S.3) Leaving the resource key null would lead to a similar problem like
> >> S.2.ADC, but there would potentially be a lot more resource keys of
> >> different logfilePaths we can't find a simple solution like the S.2.ADE
> >> for. ===============================================================
> >> 
> >> 
> >> What do you think? Did I miss something suggesting to implement S + S.1?
> >> 
> >> Kind regards
> >> Andreas :-)
> >> 
> >> On 01/03/2011 11:39 AM, Lukas Krejci wrote:
> >>> Hi,
> >>> 
> >>> The name of the resource cannot be changed programatically in RHQ<= 3
> >>> in the plugin code. The only option you have is either to change the
> >>> name manually using the UI or use the RHQ CLI and have a script that
> >>> would periodically scan for changes in the plugin configs and rename
> >>> the resources accordingly.
> >>> 
> >>> In RHQ 4 however, this is going to be possible from within your plugin
> >>> code with the resource upgrade functionality, but you're still going to
> >>> have to do a bit of manual setup to get it working.
> >>> 
> >>> You can read about the resource upgrade on our wiki:
> >>> http://www.rhq-project.org/display/RHQ/Design+-+Resource+Upgrade
> >>> 
> >>> Using the resource upgrade functionality, you are currently able to
> >>> upgrade the resource's key, name and description. But the RHQ server is
> >>> by default configured to ignore the name and description changes (so
> >>> that we don't override the custom values users might have assigned to
> >>> their resources). Currently there is no way of changing that
> >>> configuration from the UI.
> >>> 
> >>> To enable the upgrade of resource name and description, you have to
> >>> change the RHQ server system configuration using this SQL (for
> >>> Postgres):
> >>> 
> >>> INSERT INTO RHQ_SYSTEM_CONFIG (ID, PROPERTY_KEY, PROPERTY_VALUE) VALUES
> >>> (nextval('rhq_system_config_id_seq'),
> >>> 'RESOURCE_GENERIC_PROPERTIES_UPGRADE', 'true');
> >>> 
> >>> and restart the server.
> >>> 
> >>> In your plugin code, you'll have to make your discovery class implement
> >>> the ResourceUpgradeFacet interface and implement the upgrade() method
> >>> along the lines of:
> >>> 
> >>> public ResourceUpgradeReport
> >>> upgrade(ResourceUpgradeContext<HttpHostResourceComponent>   context) {
> >>> 
> >>> Configuration pluginConfig = context.getPluginConfiguration();
> >>> 
> >>> //TODO extract the values from plugin config
> >>> String hostname = null;
> >>> String port = null;
> >>> 
> >>> //TODO implement
> >>> String newResourceName = generateResourceName(hostname, port);
> >>> 
> >>> ResourceUpgradeReport report = new ResourceUpgradeReport();
> >>> report.setNewResourceName(newResourceName);
> >>> 
> >>> return report;
> >>> }
> >>> 
> >>> Lukas
> >>> 
> >>> On Tuesday, December 28, 2010 07:04:51 Lin Gao wrote:
> >>>> Hi:
> >>>>      If the value of one inventory connection is part of the resource
> >>>> 
> >>>> component name, how to make the resource component name reflecting the
> >>>> changes of the inventory connection value.
> >>>> 
> >>>>     below is an example:
> >>>> 
> >>>>     rhq-plugin.xml:
> >>>> <server name="HttpServer-Host" description="Http server host and port"
> >>>> 
> >>>>                discovery="HttpHostDiscoveryComponent"
> >>>> 
> >>>> class="HttpHostResourceComponent"
> >>>> 
> >>>>                supportsManualAdd="true">
> >>>> 
> >>>> <plugin-configuration>
> >>>> <c:simple-property name="host" description="host of target http
> >>>> server" />  <c:simple-property name="port"  description="port of
> >>>> target http server" />  </plugin-configuration>
> >>>> </server>
> >>>> 
> >>>> HttpHostDiscoveryComponent.java:
> >>>> 
> >>>> public Set<DiscoveredResourceDetails>   discoverResources(
> >>>> 
> >>>>                ResourceDiscoveryContext<HttpHostResourceComponent>
> >>>>                context) throws InvalidPluginConfigurationException,
> >>>>                Exception {
> >>>>            
> >>>>            Set<DiscoveredResourceDetails>   result = new
> >>>> 
> >>>> HashSet<DiscoveredResourceDetails>();
> >>>> 
> >>>>            for(Configuration config:
> >>>>            context.getPluginConfigurations()){
> >>>>            
> >>>>                String host =
> >>>>                config.getSimpleValue(PLUGIN_CONFIG_HOST_KEY,
> >>>> 
> >>>> null);
> >>>> 
> >>>>                Integer port =
> >>>> 
> >>>> Integer.valueOf(config.getSimpleValue(PLUGIN_CONFIG_PORT_KEY, "80"));
> >>>> 
> >>>>                String resourceKey = "HTTP_CHECK_" + host + ":" + port;
> >>>>                String resourceName = "Http Checker on: " + host + ":"
> >>>>                + port; String resourceVersion = "1.0.0";
> >>>>                String resourceDescription = "Http Checker on server: "
> >>>>                +
> >>>> 
> >>>> host + ":" + port + ".";
> >>>> 
> >>>>                DiscoveredResourceDetails resource = new
> >>>> 
> >>>> DiscoveredResourceDetails(context.getResourceType(), resourceKey,
> >>>> resourceName, resourceVersion, resourceDescription, config, null);
> >>>> 
> >>>>                result.add(resource);
> >>>>            
> >>>>            }
> >>>>            return result;
> >>>>        
> >>>>        }
> >>>> 
> >>>> If the host is defined to: 127.0.0.1 and port to: 80 when the server
> >>>> is added manually, the name of this resource component in the
> >>>> navigate tree is: "Http Checker on: 127.0.0.1:80", after that, if the
> >>>> host is changed(inventory->connection->edit...), the name of the
> >>>> resource component in the navigate tree is not changed, even re load
> >>>> the tree, so how to make it changed also?
> >>> 
> >>> _______________________________________________
> >>> rhq-devel mailing list
> >>> rhq-devel at lists.fedorahosted.org
> >>> https://fedorahosted.org/mailman/listinfo/rhq-devel
> >> 
> >> _______________________________________________
> >> rhq-devel mailing list
> >> rhq-devel at lists.fedorahosted.org
> >> https://fedorahosted.org/mailman/listinfo/rhq-devel
> > 
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/rhq-devel
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel


More information about the rhq-devel mailing list