On 07/27/2011 11:51 AM, Jason Guiditta wrote:
> On Wed, 2011-07-27 at 11:38 -0400, Christopher Alfonso wrote:
>>
>> On Jul 27, 2011, at 11:14 AM, "Hugh Brock" <hbrock@redhat.com> wrote:
>>
>>> On Wed, Jul 27, 2011 at 11:05:28AM -0400, Jason Guiditta wrote:
>>>> On Wed, 2011-07-27 at 10:43 -0400, Hugh Brock wrote:
>>>>> On Wed, Jul 27, 2011 at 10:26:08AM -0400, Jason Guiditta wrote:
>>>>>> Hello list (and Steve), I have been tasked with looking into the
>>>>>> following nebulous feature item for consideration of being included in
>>>>>> Iteration 4:
>>>>>> '* HA configuration available?'
>>>>>>
>>>>>> This is from an 'infrastructure around aeolus' perspective, if that
>>>>>> helps context at all.
>>>>>>
>>>>>> There was brief discussion of this in an extremely broad way on a recent
>>>>>> call, and given my lack of involvement with anything HA, I find myself
>>>>>> with absolutely no idea how to describe what might need doing here, let
>>>>>> alone how it might be done. I will be looking through the list archives
>>>>>> to see what has been discussed, but all the threads I recall seeing were
>>>>>> very long and detailed, and I would rather get a synopsis if someone can
>>>>>> summarize the current thinking in this area. Also, any links to specific
>>>>>> threads or documentation/designs would be much appreciated. Thanks for
>>>>>> any direction here,
>>>>>
>>>>> Sure.
>>>>>
>>>>> One of the requirements the SAs brought up is that it should be
>>>>> possible to make Conductor and its attendant pieces highly
>>>>> available. I'm not sure this is something we need to actually
>>>>> implement for release 0.4.0, but it would be nice to at least have a
>>>>> design for it by then. At a minimum we would need the entire app to be
>>>>> able to run in failover mode on two machines. We don't have to provide
>>>>> the infrastructure to deal with the failover, but the app needs to
>>>>> handle being failed over gracefully without data loss or (much)
>>>>> service interruption.
>>>>>
>>>>> Does that help at all?
>>>>>
>>>>> --Hugh
>>>>>
>>>>
>>>> If that is really the gist, then yes, it is a start, thanks. Sounds
>>>> like a feature to see how to best setup rails/$backing-db to be
>>>> clustered, and possibly separate tasks to research same some some of the
>>>> other components (iwhd seems a likely candidate). However, I have this
>>>> feeling there is much more to it than that. Perhaps this task/feature
>>>> is a combination of documenting how to do some of these thing, and
>>>> writing up what what does and does not currently exist for this kind of
>>>> scenario? Also, probably something around 'what is going to be the
>>>> aeolus story for HA, and what does it mean?'.
>>>
>>> If all we got done was that, I'd be delighted.
>>>
>>> --H
>>>
>>> --
>>> == Hugh Brock, hbrock@redhat.com ==
>>> == Engineering Manager, Cloud BU ==
>>> == Aeolus Project: Manage virtual infrastructure across clouds. ==
>>> == http://aeolusproject.org ==
>>>
>>> "I know that you believe you understand what you think I said, but I’m
>>> not sure you realize that what you heard is not what I meant."
>>> --Robert McCloskey
>>> _______________________________________________
>>> aeolus-devel mailing list
>>> aeolus-devel@lists.fedorahosted.org
>>> https://fedorahosted.org/mailman/listinfo/aeolus-devel
>>
>> If there is interest in a easy way to implement an HA database
>> configuration and you don't mind pushing users to a MySQL based
>> solution, I'll send out the m2s and m2m configs and describe how we're
>> flipping back and forth between nodes. access.redhat.com/management is
>> using master to master, and it's working out quite well. Let me know.
>>
>> Chris
>
> Even though our official supported db is postgres, any HA docs of any
> kind would be much appreciated - we may make them optional upstream info
> on our wiki or something if you are not opposed.
>
> _______________________________________________
> aeolus-devel mailing list
> aeolus-devel@lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/aeolus-devel

If you're familiar with authoring/maintaining puppet modules you can jump straight to the referenced docs [1,2] and drill down through the puppet manifests.  If not, I'll describe what's going on here, and probably what you would really be interested in.

Keep in mind, for the candlepin databases that sit behind access.redhat.com/management we have a VIP sitting in front of two MySQL m2m configured databases.  If there is a connectivity problem detected at the database VIP at runtime, connections from the application server switch from the active master to the standby master.  If no VIP sits between the application server and the multiple m2m MySQL nodes, it's a bit more challenging to automatically fail over (you get to implement the retry yourself!).

The candlepin databases in production are set up using the following yaml files (which just reference which puppet classes to pull in, the class you're interested in is candlepin:db).  That module is defined in http://git.corp.redhat.com/cgit/puppet-cfg/modules/candlepin/tree/manifests/db.pp.  You can follow the references to the puppet classes that are invoked, but the important thing to take away from this is that each MySQL node is setup as a master and a slave, such that node A is a slave to node B, and node B is a slave to node A (hence the master to master replication).

Node 1: http://git.corp.redhat.com/cgit/puppet-cfg/manifests/tree/nodes/db02.candlepin.prod.ext.phx2.redhat.com/init.yaml
Node 2: http://git.corp.redhat.com/cgit/puppet-cfg/manifests/tree/nodes/db01.candlepin.prod.ext.phx2.redhat.com/init.yaml

There is a bit of automation in place and custom puppet types in the libmysql module, but at the end of the day, the nodes are configured with http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/tree/templates/cnf/master.erb and http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/tree/templates/cnf/slave.erb (values filled in from the puppet manifests).

Once the configuration files are laid down and mysqld is running, running the CHANGE MASTER TO[3] syntax will 'wire' the nodes together.

References:
[1] https://docspace.corp.redhat.com/docs/DOC-59614
[2] http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/
[3] http://dev.mysql.com/doc/refman/5.0/en/change-master-to.html

Thanks!

Chris