David Means and I have been working on putting together a
specification/requirement list for what we need for the mirror
management task. This is for two reasons. One, we might not get all of
the Centos pieces in a timely matter, so we will move forward with our
own code. Two, if we get all the pieces we need to make sure they are
going to work as needed. In an effort to complete the list, I am asking
for some guidance in understanding the process as a whole. Please look
over the list below and make any necessary comments. Here is our
* Cron job to check all of the mirrors for freshness. We got this piece
from Seth(Centos team - makemirrorlists2-now.pl), but it is going to need
* Database dictionary - What kind of data do we need to store for each
* We will need to maintain a list of mirror sites and their state (ie
useable, stall, not reachable, disabled.)
* Some kind of front end to redirect clients to their best mirror match(ie
* Admin script to manage mirror list(additions, modifications, deletions,
* Once a yum session starts, we will need to direct all of the requests
for that update session to the *same* mirror. That way the updates will
be consistent, even if not to the most recent version. If this is true
then this increases the complexity of the task, because we need to a.
notice when a new session begins, and record the IP source. b. check each
incoming request against the known table of ongoing sessions, and send
continuing requests on to the assigned mirror. c. clean up the sessions
table when the update session ends.
* Handle sites that come back with a new IP, after being off the network.
Has the site name been hijacked or just part of some network clean up
David, if I missed anything, please correct me. As for the rest of you,
please look over the list and reply with any amount of insight.
I just now had a chance to go over the meeting notes from last week and
I noticed the discussion on TG. In FC5 it is included in the extras
repo, but I had problems getting through the wiki tutorial. I seem to
remember it had something to with SQLObjects having some issues. After
couple of hours of messing around with it I decided to use the
ez_setup.py script. After using ez_setup.py script to install TG
everything work as it should. The ez_setup.py uses Python eggs, which
so far have worked well for me. In any case, I think TG is great!
Thought I would take a moment to introduce myself. My name is David
Eisenstein. I am a volunteer with the Fedora Legacy project
at this time, and have recently started also helping out the fine Fedora
I have an interest in helping out with the Fedora Infrastructure project.
What I am most keenly interested in at this time is being here to help work
with the upcoming Fedora Legacy integration that has been blessed by the
and to help in continuing maintenance of such (new?) systems/computers/
I am also interested in helping to do whatever I can to help in lowering
the bar for new sub-projects to take root. And in learning stuff to become
Regarding Legacy, I like the idea of working in Fedora's CVS system,
particularly if already-existing information in cvs.fedora.redhat.com about
software in legacy or to-be-legacy Fedora releases can be extracted to
pre-populate the area(s) that Legacy would use in cvs.fedoraproject.org for
our software maintenance activities.
I am not sure what planning has been formally or informally done yet other
* Jesse Keating's plan about this Integration:
* Elliot Lee's response on the FAB list, here
Jesse, Elliot, has anything developed on Legacy Integration since these
writings? Do we have any kind of sketchy timetable in mind for any of
these activities? Or would we want to establish any timetables at this
is there any public bunch of computers where upstream developers can
test their applications on Fedora? I know about upstream developers
who don't use Linux as primary OS or they use different distributions
and test upstream code with things like SELinux is difficult without
access to machine with useful FC.
Karel Zak <kzak(a)redhat.com>
The webmaster(a)fedoraproject.org account is now part of the ticketing
system. Emails sent to webmaster(a)fedoraproject.org will automatically
get created in the webmaster queue. Users who would like access to the
system to answer emails on behalf of the webmaster email address please
see the web address that should be printed below.
This is the first account we've put in the OTRS like this. Please let
us know what you guys think of it because eventually we'd like to move
other accounts into the system.
As a test I'd like someone in the web-members group who already has
access to the ticketing system to lock it and close it.
Hey guys, thought I'd start a backup thread. One thing I'd like to
start doing is backing up the home directories, I've had my home dir
blown out a couple of times that resulted in script loss. Same thing
happened recently to some of the FESCo candidates who had their access
revoked to ensure a fair election :-D.
Refer to previous meeting notes and list emails for whats been said so
far but here's the short list:
bacula, amanda, rsync and scrips, rdiff-backup
dgilmore is one of the guys leading the backup stuffbut participation is
encouraged by everyone. I'd like to throw backuppc in the list just
because I use it / like it. It might be good for our environment. I
get the sense that bacula is the current front runner. One thing should
be noted. Whatever system we go with needs to be in Extras or Core so
we can guarantee ongoing support. But lots of good products have gotten
into our repos this way (Nagios :-P).
I'd prefer whatever we choose to have native ssh tunnel support. if this
isn't possible we can always create the tunnels manually. For those
that aren't familiar we have 2 primary sites, one in Phoenix and one at
Duke. These two sites are somewhat linked but still pretty separate.
Any thoughts / suggestions?
I've been corresponding off-list with Elliot with regard to upgrading
db1. At his suggestion, I'm going to run my proposed process by
everyone for feedback. I went through a production DB server upgrade a
while back and this process reflects what I did then and lessons learned
during the process.
I also had Slony in this mix, which resulted in a few more steps. From
what I've read on the wiki pages regarding the current Fedora
infrastructure setup, there is only 1 DB server, a SPOF. It may be
worth the time to start a separate conversation about some sort of
replication or clustered setup, which a beast unto itself.
I have a master/slave replication setup with Slony which works
quite well. You still have the SPOF with the master but at least you
can load balance between the slaves and still have read access going if
the master goes down. With all of the research I've done, I've not
found anything which is production ready and allows a multi master
scenario for Postgres, only master/slave scenario with replication, like
Slony. The closest thing I found was PGCluster but it was very flaky,
at least on 8.x, the last time I played with it late last year.
PGCluster was also very hardware intensive, meaning it takes like 4
machines to run a setup without a SPOF.
The following assumes the desire for as minimal as possible downtime and
as a result is a little more involved than if more downtime were
tolerable. I welcome comments and refinements to the process.
Now, onto the upgrade process.
*** Make a DB and filesystem backup ***
Ensure that all necessary config files are archived so they can be
quickly reinstalled via kickstart later
1) Setup all apps to connect to DB using a host alias in /etc/hosts
2) Setup another server as a temporary DB server
Note: An option here is to copy data to the temp DB server and do an
immediate cut over to the temp server after the data has been copied but
you run the risk of data being out of sync between the master and temp
server. Danger, Will Robinson!
3) Temporarily disable apps hitting DB
4) pg_dump DBs and template0, users, groups, etc over to temp server
5) Test to ensure DB is up and accepting connections
6) Change the alias entry in /etc/hosts on hosts running apps to the
record for the temp DB server
7) Re-enable apps
8) Re-install and configure OS on old DB server via kickstart
9) Temporarily disable apps hitting DB
10) pg_dump DBs and template0, users, groups, etc over to master server
11) Test to ensure master DB is up and accepting connections
12) Change the alias entry in /etc/hosts on servers running apps back to
the record for the master DB server
There, my 12 step program for a DB upgrade. :-)
I had a few more Slony related steps which required shuffling the app
servers between Slony slaves and the master but the above it basically
I made a few decisions/assumptions during my process.
In my case since 99% of DB hits were reads,I still had my Slony slaves
accepting read requests. I was able to do this since I maintain both
read and write DB handles in the SoftSwitch. Some apps allow a
"degraded" mode where data is "read only" just for this sort of thing,
I'm not sure if this is the case here or not.
During the time I disabled the apps to copy over the DB to the temp
server, users just couldn't login to the web portal to change their
settings, which are written to the master, for a few minutes. I had to
ask, would this really be an issue for 5 min at 3am CST on a Sunday
morning? (assuming that the DBs can be copied in the span of 5 min, I
have a VLAN'd GigE management network for this sort of thing) The same
downtime was experienced when I switched back over to the master DB.
Since I'm assuming we're upgrading from Postgres 7.x we'll definitely
want to do a pg_dump/pg_restore since there are some inherent
differences in the data structures on the disk. In the past I've just
stopped postgres, copied over /var/lib/pgsql to the new server and
started postgres there, and called it good but you can't do that when
upgrading from 7.x to 8.x. Since the data itself has to be migrated
from the 7.x format to the 8.x format at some point, there is probably
going to have to be some measure of downtime.
Otherwise you get into having to compare data in 2 different databases
to see if anything changed and manually replicating those changes to the
master copy. For me, the 5 min of partial downtime was _much_ cheaper.
I look forward to discussion on this.
It seems that pass-through ssh key authentication isn't working anymore.
Was my account disabled on fedora.linux.duke.edu due the Extras
election too, or is this some other issue?
On Fri, June 23, 2006 7:30 pm, Curt Moore wrote:
> Hi Jason.
> At the suggestion of Elliot Lee, I'm contacting you about the Mirror
> Management project.
> I'd like to help so please let me know what I can do.
> I saw that there was talk about borrowing the CentOS mirroring product.
> Has anyone over at CentOS been contacted about this? I used to work
> with some of the CentOS maintainers, but haven't spoken with them in a
> while, but it may be an "in".
> Please let me know where things stand.
Seth Vidal has been working with Lance from the Centos team on getting
the Centos mirroring code. If know or would like to contact Lance about
the code, I am sure Seth would not mind. Currently we have the cron job
that runs to see what mirror sites are stall. Here is the break down of
It's broken up into 3 parts:
1. one part queries their mysql database of the mirrors and builds a
mirrorlist using geoip
2. another part is a cron job that connects to the mirrors and grabs
the repomd.xml off of them to determine 'freshness'
3. the last part generates the web page based on the database info
In addition to the 3 parts, we also need the table structure of their
Thanks, for your help!