deploying additional storage nodes

John Sanda jsanda at redhat.com
Thu Jul 18 20:10:23 UTC 2013


On Jul 18, 2013, at 3:47 PM, Stefan Negrea <snegrea at redhat.com> wrote:

> 
> 
> ----- Original Message -----
>> From: "Jay Shaughnessy" <jshaughn at redhat.com>
>> To: rhq-devel at lists.fedorahosted.org
>> Sent: Thursday, July 18, 2013 2:07:59 PM
>> Subject: Re: deploying additional storage nodes
>> 
>> 
>> On 7/18/2013 2:46 PM, Stefan Negrea wrote:
>>> 
>>> ----- Original Message -----
>>>> From: "Jay Shaughnessy" <jshaughn at redhat.com>
>>>> To: rhq-devel at lists.fedorahosted.org
>>>> Sent: Thursday, July 18, 2013 10:20:57 AM
>>>> Subject: Re: deploying additional storage nodes
>>>> 
>>>> 
>>>> After thinking about this I think I prefer option 3, which delays
>>>> incorporating the new storage node until after discovery.  This
>>>> eliminates the need for storage node installation to talk to the
>>>> database directly, taking away those "cons".   It also eliminates the
>>>> need to contact a running server, which means the servers don't need to
>>>> be up (this whole thing may be part of a maintenance period), and we
>>>> don't need to deal with establishing a connection (failure point),
>>>> getting the host/port/creds (new cmd line options and failure point).
>>>> 
>>>> In option 3 we can actually fail to install the new storage node, revert
>>>> the new storage node install, fix issues with agent connectivity, or
>>>> whatever, without immediately mucking with the RHQ topology.  It defers
>>>> integrating the storage node until 1) C* is actually running, 2) The
>>>> required agent is actually running, 3) the server is running, and 4) the
>>>> storage node is discovered and merged into inventory.
>>> I think this is the best argument so far in favour of any of the options.
>>> Also, option 3 is more aligned with our proposed architecture where
>>> complete management of storage nodes is done exclusively via agents.
>>> Making even the install configuration follow the same pattern is just
>>> enforcing that decision.
>>> 
>>> However, the one thing that I did not yet see mentioned for option 3 is the
>>> initial install. This process would be completely different because there
>>> is no server the agent can connect to receive storage node configuration.
>>> So one of the major cons from option 2 applies to option 3 too; and that
>>> problem will be even more visible in option 3 than it would be in option
>>> 2.
>>> 
>>> So, I am inclined to side with option 3 if we can find a clean and simple
>>> approach for the initial storage node installation.
>> 
>> The initial install would be as today, I think.  The RHQ Server
>> installer would notice that there were no storage node entries in the
>> DB.  As such it would find the storage node via the rhq.cassandra.seeds
>> property in the rhq-server.properties file, make the initial definition
>> and connect to the configured storage node. If using 'rhqctl install'
>> then the storage node installer would have already updated the
>> rhq-server.properties file with the co-located storage node seed entry.
>> If using 'rhqctl install --server' then it would have been up to the
>> user to manually enter the rhq.cassandra.seeds property with the one or
>> more [previously] installed RHQ Storage Nodes.
>> 
>> The server installer already has a connection to the DB, so it can
>> perform the necessary manipulation for the storage node entries. When
>> the storage node is finally discovered and reported by the agent, it
>> will already have an entry, and I think, can just be auto-imported and
>> that's it.  For subsequent storage node additions the server installer
>> would either not have been run (storage node only) or would ignore the
>> rhq.cassandra.seeds entry, deferring to what is described in option 3.
>> 
>> Is that a reasonable approach?
>> 
> 
> The first problem with this approach is the assumption that first storage node will always be collocated with the RHQ Server. Another problem, option 3 assumes that the installation of an RHQ storage node leaves the storage node is an un-configured. So maybe if we can define better what will be configured out of box (and what not) will make the initial install process more obvious. It would not be a good idea to have two separate configuration paths in two parts of the code (one for the initial install and one or adding new nodes). It will just increase complexity which is my primary concern with this option.
> 

What Jay said here is correct. It does not assume that the storage node and server are co-located. There are instructions on the wiki for installing them separately on different machines. The server cannot and should not provide the configuration for initial storage node. That is provided by the user either explicitly by setting storage installer options or implicitly by using default values for those options. Once that initial storage node is installed, it is configured. The issue isn't with the initial node, it is with subsequent nodes because once those cluster-wide settings are established with the initial node, they need to be the same for subsequent nodes. There are two configuration paths because there are two different scenarios to handle. In one scenario with the initial node, it is completely reasonable to allow the user to choose non-default values for the cql, jmx, and storage (i.e., gossip) ports. But for subsequent nodes, those port values have to be the same. 

One thing that has not been addressed in this thread is deploying multiple storage nodes before installing the RHQ server. This would be an advanced approach more suitable for larger deployments that know up front that they want to run multiple storage nodes. In this situation, the user will have to make sure he configures each node correctly.


More information about the rhq-devel mailing list