On Thu, 10 Nov 2011 14:45:16 -0700
Kevin Fenzi <kevin(a)scrye.com> wrote:
I talked about this at todays meeting, but I thought I would note it
here for further input and also see if there was interest in doing a
Virtual Fedora Activity Day next week to finish out hosted plans.
background:
We have 5 servers at serverbeach (Thanks for donating them SB Folks!)
They were setup long ago and are getting a bit old in the tooth, so we
are getting 5 new machines to migrate our stuff to. This will allow us
to be on hardware thats newer/better supported and also move to new OS
and re-arrange things. We must migrate to those new machines by
2011-12-11. (around a month from now).
current:
serverbeach01 -> a mirror
serverbeach02 -> collab01 (all our
lists.fedoraproject.org lists).
ns01
serverbeach03 -> collab02 (a warm spare of collab01, it rsyncs from
there every hour).
serverbeach04 -> hosted01 (all
fedorahosted.org)
serverbeach05 -> hosted02 (a warm spare, but currently down and not
working).
Plan:
serverbeach06 -> collab03
ns01
serverbeach07 -> collab04
(move these to rhel6 for both host/guest. Also, look at setting up
drbd for /srv on them, so we can easily switch over from 03 to 04 in
case of failure. If we do do drbd, I'd like to schedule at least one
failover to make sure it really is all working as expected. ns01 can
go on either. ) We will also need to add drbd rpms to our repo.
That leaves us with hosted to deal with, and serverbeach08/09/10 to do
it in (currently). We have discussed lots of plans with hosted, I'd
like to propose meeting up on IRC next Wed to do a virtual Fedora
Activity Day around planning that. Or we could just try and work out
details here on list.
Things we have discussed:
* Moving
lists.fedorahosted.org to collab*
* Moving projects to a CNAME or aliased setup so we can then move them
to whatever backend host we like. (what exact changes?)
* Making whatever changes we need now so we could move to spinning out
some projects into "the cloud" instances. What are those specific
changes?
Issues:
* Backups of hosted01 take ~6 hours, which is not very good. ;( (thats
incrementals, full backups are much slower... days).
* Hosted01 is pretty heavily loaded. Some of which may just disappear
when we move to rhel6 (I know some of it will).
* Ideally we could bring up projects on a new instance in the event of
a failure of a node.
I'd like to have a SOP with steps for migrating hosted instances soon.
1. I think we need some volunteer projects. Do we know of any
enterprising folks who would like to be suckers.... err test subjects?
2. For collab2 and hosted2 - would it be possible to run both of
those secondary machines on one of the new sb boxes? Disk-space wise we
should be able to make it work - and it from a failover standpoint it
feels unlikely we would need to failover both hosted1 and collab1 at
the same time.
3. Then do backups from collab2 and hosted2 - until we have our stuff
together for hosted-tng.
So, we can say, 'here's what we are going to change and
why'. Then we
can get setup and start moving projects that opt-in, then have a flag
day before our migration window to just move everyone else.
I suspect we'll need to waddle them all through a little at a time.
Remember getting rid of cvs.fedoraproject.org? like that :(
-sv