On 07/27/2011 11:51 AM, Jason Guiditta wrote:
On Wed, 2011-07-27 at 11:38 -0400, Christopher Alfonso wrote:
>
> On Jul 27, 2011, at 11:14 AM, "Hugh Brock" <hbrock(a)redhat.com>
wrote:
>
>> On Wed, Jul 27, 2011 at 11:05:28AM -0400, Jason Guiditta wrote:
>>> On Wed, 2011-07-27 at 10:43 -0400, Hugh Brock wrote:
>>>> On Wed, Jul 27, 2011 at 10:26:08AM -0400, Jason Guiditta wrote:
>>>>> Hello list (and Steve), I have been tasked with looking into the
>>>>> following nebulous feature item for consideration of being included
in
>>>>> Iteration 4:
>>>>> '* HA configuration available?'
>>>>>
>>>>> This is from an 'infrastructure around aeolus' perspective,
if that
>>>>> helps context at all.
>>>>>
>>>>> There was brief discussion of this in an extremely broad way on a
recent
>>>>> call, and given my lack of involvement with
anything HA, I find myself
>>>>> with absolutely no idea how to describe what might need doing
here, let
>>>>> alone how it might be done. I will be looking
through the list
archives
>>>>> to see what has been discussed, but all the
threads I recall
seeing were
>>>>> very long and detailed, and I would rather get a
synopsis if
someone can
>>>>> summarize the current thinking in this area.
Also, any links to
specific
>>>>> threads or documentation/designs would be much
appreciated. Thanks for
>>>>> any direction here,
>>>>
>>>> Sure.
>>>>
>>>> One of the requirements the SAs brought up is that it should be
>>>> possible to make Conductor and its attendant pieces highly
>>>> available. I'm not sure this is something we need to actually
>>>> implement for release 0.4.0, but it would be nice to at least have a
>>>> design for it by then. At a minimum we would need the entire app to be
>>>> able to run in failover mode on two machines. We don't have to
provide
>>>> the infrastructure to deal with the failover, but the app needs to
>>>> handle being failed over gracefully without data loss or (much)
>>>> service interruption.
>>>>
>>>> Does that help at all?
>>>>
>>>> --Hugh
>>>>
>>>
>>> If that is really the gist, then yes, it is a start, thanks. Sounds
>>> like a feature to see how to best setup rails/$backing-db to be
>>> clustered, and possibly separate tasks to research same some some of the
>>> other components (iwhd seems a likely candidate). However, I have this
>>> feeling there is much more to it than that. Perhaps this task/feature
>>> is a combination of documenting how to do some of these thing, and
>>> writing up what what does and does not currently exist for this kind of
>>> scenario? Also, probably something around 'what is going to be the
>>> aeolus story for HA, and what does it mean?'.
>>
>> If all we got done was that, I'd be delighted.
>>
>> --H
>>
>> --
>> == Hugh Brock, hbrock(a)redhat.com ==
>> == Engineering Manager, Cloud BU ==
>> == Aeolus Project: Manage virtual infrastructure across clouds. ==
>> ==
http://aeolusproject.org ==
>>
>> "I know that you believe you understand what you think I said, but I’m
>> not sure you realize that what you heard is not what I meant."
>> --Robert McCloskey
>> _______________________________________________
>> aeolus-devel mailing list
>> aeolus-devel(a)lists.fedorahosted.org
>>
https://fedorahosted.org/mailman/listinfo/aeolus-devel
>
> If there is interest in a easy way to implement an HA database
> configuration and you don't mind pushing users to a MySQL based
> solution, I'll send out the m2s and m2m configs and describe how we're
> flipping back and forth between nodes.
access.redhat.com/management is
> using master to master, and it's working out quite well. Let me know.
>
> Chris
Even though our official supported db is postgres, any HA docs of any
kind would be much appreciated - we may make them optional upstream info
on our wiki or something if you are not opposed.
_______________________________________________
aeolus-devel mailing list
aeolus-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/aeolus-devel If you're familiar with
authoring/maintaining puppet modules you can
jump straight to the referenced docs [1,2] and drill down through the
puppet manifests. If not, I'll describe what's going on here, and
probably what you would really be interested in.
Keep in mind, for the candlepin databases that sit behind
access.redhat.com/management we have a VIP sitting in front of two MySQL
m2m configured databases. If there is a connectivity problem detected
at the database VIP at runtime, connections from the application server
switch from the active master to the standby master. If no VIP sits
between the application server and the multiple m2m MySQL nodes, it's a
bit more challenging to automatically fail over (you get to implement
the retry yourself!).
The candlepin databases in production are set up using the following
yaml files (which just reference which puppet classes to pull in, the
class you're interested in is candlepin:db). That module is defined in
http://git.corp.redhat.com/cgit/puppet-cfg/modules/candlepin/tree/manifes....
You can follow the references to the puppet classes that are invoked,
but the important thing to take away from this is that each MySQL node
is setup as a master and a slave, such that node A is a slave to node B,
and node B is a slave to node A (hence the master to master replication).
Node 1:
http://git.corp.redhat.com/cgit/puppet-cfg/manifests/tree/nodes/db02.cand...
Node 2:
http://git.corp.redhat.com/cgit/puppet-cfg/manifests/tree/nodes/db01.cand...
There is a bit of automation in place and custom puppet types in the
libmysql module, but at the end of the day, the nodes are configured
with
http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/tree/template...
and
http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/tree/template...
(values filled in from the puppet manifests).
Once the configuration files are laid down and mysqld is running,
running the CHANGE MASTER TO[3] syntax will 'wire' the nodes together.
References:
[1]
https://docspace.corp.redhat.com/docs/DOC-59614
[2]
http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/
[3]
http://dev.mysql.com/doc/refman/5.0/en/change-master-to.html
Thanks!
Chris