On 07/27/2011 11:51 AM, Jason Guiditta wrote:
> On Wed, 2011-07-27 at 11:38
-0400, Christopher Alfonso wrote:
>>
>> On Jul 27, 2011, at 11:14 AM, "Hugh Brock"
<hbrock@redhat.com> wrote:
>>
>>> On Wed, Jul 27, 2011 at 11:05:28AM -0400, Jason
Guiditta wrote:
>>>> On Wed, 2011-07-27 at 10:43 -0400, Hugh Brock
wrote:
>>>>> On Wed, Jul 27, 2011 at 10:26:08AM -0400,
Jason Guiditta wrote:
>>>>>> Hello list (and Steve), I have been
tasked with looking into the
>>>>>> following nebulous feature item for
consideration of being included in
>>>>>> Iteration 4:
>>>>>> '* HA configuration available?'
>>>>>>
>>>>>> This is from an 'infrastructure around
aeolus' perspective, if that
>>>>>> helps context at all.
>>>>>>
>>>>>> There was brief discussion of this in an
extremely broad way on a recent
>>>>>> call, and given my lack of involvement
with anything HA, I find myself
>>>>>> with absolutely no idea how to describe
what might need doing here, let
>>>>>> alone how it might be done. I will be
looking through the list archives
>>>>>> to see what has been discussed, but all
the threads I recall seeing were
>>>>>> very long and detailed, and I would
rather get a synopsis if someone can
>>>>>> summarize the current thinking in this
area. Also, any links to specific
>>>>>> threads or documentation/designs would be
much appreciated. Thanks for
>>>>>> any direction here,
>>>>>
>>>>> Sure.
>>>>>
>>>>> One of the requirements the SAs brought up is
that it should be
>>>>> possible to make Conductor and its attendant
pieces highly
>>>>> available. I'm not sure this is something we
need to actually
>>>>> implement for release 0.4.0, but it would be
nice to at least have a
>>>>> design for it by then. At a minimum we would
need the entire app to be
>>>>> able to run in failover mode on two machines.
We don't have to provide
>>>>> the infrastructure to deal with the failover,
but the app needs to
>>>>> handle being failed over gracefully without
data loss or (much)
>>>>> service interruption.
>>>>>
>>>>> Does that help at all?
>>>>>
>>>>> --Hugh
>>>>>
>>>>
>>>> If that is really the gist, then yes, it is a
start, thanks. Sounds
>>>> like a feature to see how to best setup
rails/$backing-db to be
>>>> clustered, and possibly separate tasks to
research same some some of the
>>>> other components (iwhd seems a likely candidate).
However, I have this
>>>> feeling there is much more to it than that.
Perhaps this task/feature
>>>> is a combination of documenting how to do some of
these thing, and
>>>> writing up what what does and does not currently
exist for this kind of
>>>> scenario? Also, probably something around 'what
is going to be the
>>>> aeolus story for HA, and what does it mean?'.
>>>
>>> If all we got done was that, I'd be delighted.
>>>
>>> --H
>>>
>>> --
>>> == Hugh Brock, hbrock@redhat.com ==
>>> == Engineering Manager, Cloud BU ==
>>> == Aeolus Project: Manage virtual infrastructure
across clouds. ==
>>> == http://aeolusproject.org ==
>>>
>>> "I know that you believe you understand what you
think I said, but I’m
>>> not sure you realize that what you heard is not what
I meant."
>>> --Robert McCloskey
>>> _______________________________________________
>>> aeolus-devel mailing list
>>> aeolus-devel@lists.fedorahosted.org
>>>
https://fedorahosted.org/mailman/listinfo/aeolus-devel
>>
>> If there is interest in a easy way to implement an HA
database
>> configuration and you don't mind pushing users to a MySQL
based
>> solution, I'll send out the m2s and m2m configs and
describe how we're
>> flipping back and forth between nodes.
access.redhat.com/management is
>> using master to master, and it's working out quite well.
Let me know.
>>
>> Chris
>
> Even though our official supported db is postgres, any HA
docs of any
> kind would be much appreciated - we may make them optional
upstream info
> on our wiki or something if you are not opposed.
>
> _______________________________________________
> aeolus-devel mailing list
> aeolus-devel@lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/aeolus-devel
If you're familiar with authoring/maintaining puppet modules you can
jump straight to the referenced docs [1,2] and drill down through
the puppet manifests. If not, I'll describe what's going on here,
and probably what you would really be interested in.
Keep in mind, for the candlepin databases that sit behind
access.redhat.com/management we have a VIP sitting in front of two
MySQL m2m configured databases. If there is a connectivity problem
detected at the database VIP at runtime, connections from the
application server switch from the active master to the standby
master. If no VIP sits between the application server and the
multiple m2m MySQL nodes, it's a bit more challenging to
automatically fail over (you get to implement the retry yourself!).
The candlepin databases in production are set up using the following
yaml files (which just reference which puppet classes to pull in,
the class you're interested in is candlepin:db). That module is
defined in
http://git.corp.redhat.com/cgit/puppet-cfg/modules/candlepin/tree/manifests/db.pp.
You can follow the references to the puppet classes that are
invoked, but the important thing to take away from this is that each
MySQL node is setup as a master and a slave, such that node A is a
slave to node B, and node B is a slave to node A (hence the master
to master replication).
Node 1:
http://git.corp.redhat.com/cgit/puppet-cfg/manifests/tree/nodes/db02.candlepin.prod.ext.phx2.redhat.com/init.yaml
Node 2:
http://git.corp.redhat.com/cgit/puppet-cfg/manifests/tree/nodes/db01.candlepin.prod.ext.phx2.redhat.com/init.yaml
There is a bit of automation in place and custom puppet types in the
libmysql module, but at the end of the day, the nodes are configured
with
http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/tree/templates/cnf/master.erb
and
http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/tree/templates/cnf/slave.erb
(values filled in from the puppet manifests).
Once the configuration files are laid down and mysqld is running,
running the CHANGE MASTER TO[3] syntax will 'wire' the nodes
together.
References:
[1] https://docspace.corp.redhat.com/docs/DOC-59614
[2] http://git.corp.redhat.com/cgit/puppet-cfg/modules/libmysql/
[3] http://dev.mysql.com/doc/refman/5.0/en/change-master-to.html
Thanks!
Chris