On 4/7/11 4:50 AM, Steve Millidge wrote:
> From looking at the posts you mention they seem to relate to new
Amazon
instances configured with instance-store having the "incorrect" agent names
when you terminate and restart them. This obviously has implications as you
have to decide whether these are new machines or old machines which have
been switched back on, in your handling of them in EC2.
If they are "new" machines then you will have lots on new servers appearing
in your inventory.
If you consider them as old machines which have just been turned on then I
think this could easily be solved by passing the agent name to the instance
at instance creation using something like cloud-init
https://help.ubuntu.com/community/CloudInit to set the agent name and
security token before it registers with the server. In the forum post the
person seems to suggest this is too much effort, in which case he has a
problem. However it is pretty easy to do, all you need to do is maintain a
map of agent names to tokens and pass these to the instance on boot, we keep
this mapping on the RHQ server as that is the server starting instances. In
fact I'm sure the server is maintaining this somewhere but we haven't
figured out where yet.
You could also configure your dynamic cluster using EBS backed instances in
which case all this faffing around goes away as these instances can be
"switched off" without losing the agent registration data. The additional
cost of EBS backed instances is pretty small when they are switched off and
the benefits of the persistent configuration outweighs the cost in my
opinion.
If on the other hand you do not have a very dynamic population of instance
store instances to manage then using RHQ has no issues.
Thanks
Steve
http://community.jboss.org/message/563562
http://community.jboss.org/message/562468
_______________________________________________
Most of the investigation I did
involved EBS images. There were two big
issues that had to be addressed. First involves resources keys. Resource
keys are supposed to be static in nature. They should not change. The
resource key for platforms and agents is based off host names and/or IP
addresses by default. When you stop an EBS machine, you wind up with a
new machine at least in so far as it has a different host name and IP
address, but the machine is backed by the same EBS volume. I want to say
that the same behavior holds when applying a restart operation, but I
have to double check. So we wind up with bad resource keys if we use the
defaults. To work around this, I used instance ids for the resource keys
since those do not change across restarts.
The second and more interesting issue involves agent/server
communication. When an agent registers with the server, the agent sends
its IP address to the server so the server knows how to contact the
agent. And the agent receives a failover list from the server that
contains a list of server host names/IP addresses. When an agent machine
is restarted, the server no longer has a valid address for the agent;
consequently, the server will not be able to contact the agent.
Likewise, if the server machine is restarted, the agent will not be able
to contact the server.
I am not familiar with CloudInit so I cannot comment on how effective it
would be at solving these problems, but there are a number of ways to
deal with these issues. I was looking for a generic solution though that
would be cloud provider agnostic. Fortunately, I was able to implement a
solution utilizing only existing RHQ infrastructure. The work I did is
in master. It has gone through some preliminary testings but nothing
extensive yet.
- John