Also avaliable at:
20:00 < mmcgrath> #startmeeting
20:00 < fedbot> Meeting started Thu Jun 11 20:00:09 2009 UTC. The chair is
20:00 < fedbot> Information about MeetBot at http://wiki.debian.org/MeetBot
Commands: #action #agreed #halp #info #idea #link #topic.
20:00 * ricky
20:00 < mmcgrath> #topic Infrastructure -- Who's here?
20:00 -!- ChitleshGoorah1 [n=chitlesh(a)d54C48C73.access.telenet.be] has joined
20:00 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Who's
20:00 * ricky
20:00 * LinuxCode
20:00 * SmootherFrOgZ is
20:00 * lmacken
20:00 * mmcgrath is here
20:00 * nirik is lurking around.
20:01 < mmcgrath> Ok, well lets get started
20:01 < mmcgrath> #topic Infrastructure -- F11 Release
20:01 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- F11 Release
20:01 -!- sijis [n=sijis(a)adsl-70-131-122-68.dsl.emhril.sbcglobal.net] has joined
20:01 < mmcgrath> .ticket 1395
20:01 < zodbot> mmcgrath: #1395 (Lessons Learned) - Fedora Infrastructure - Trac -
20:01 < mmcgrath> So what went wrong with this release?
20:02 < mmcgrath> All in all I think it went very well.
20:02 < nirik> very little I think. ;)
20:02 < mmcgrath> The early bit flip (thanks f13) provided a 75% mirror rate when
the anouncement went out.
20:02 * herlo is lurking
20:02 < mmcgrath> thats much better then for F10.
20:02 < ricky> I didn't expect docs links to change, so we had to do some
last-minute updates that made us a bit late
20:02 < ricky> We'll make sure to double check that better next time around.
20:02 -!- ChitleshGoorah2 [n=chitlesh(a)123.5-241-81.adsl-dyn.isp.belgacom.be] has joined
20:02 < mmcgrath> Also, as mdomsch has pointed out many times, the torrents were
screaming. The fastest torrent dl's I've ever had during a release.
20:03 * sijis is here.
20:03 < mmcgrath> 15-20 minutes for 4 isos compared to 8 or 9 hours previously.
20:03 * ianweller
20:03 * abadger1999 here
20:03 < nirik> yeah, due to them opening early we got a lot more seeders at release
20:03 -!- ldimagg__ [n=ldimaggi(a)pool-96-252-122-35.bstnma.fios.verizon.net] has joined
20:03 < mmcgrath> ricky: should we make that a docs task or an us task?
20:03 < ricky> That'll be a websites task, I'll send links to the test page
to list in advance.
20:04 * johe_ here too, by the way
20:04 < mmcgrath> k
20:04 < mmcgrath> The other thing that went kind of wacky was the bit leaks we had
the thursday or friday before hand.
20:04 < mmcgrath> I think f13 is working on those though.
20:05 < mmcgrath> Oh, one other thing.
20:05 < mmcgrath> Proxy3 was hurting.
20:05 < mmcgrath> And I think I know why but I need to do a bit more research.
20:05 < mmcgrath> long story short it was doing too much.
20:05 < mmcgrath> I think it's just because it's one of our slower boxes in
terms of cpu and disk.
20:05 < ricky> Did the x86_64 vs. i386 difference also come into play?
20:06 < mmcgrath> ricky: not this time. But proxy1 and proxy2 are kind of odd
because they're behind a balancer.
20:06 * mmcgrath needs to fix that.
20:06 < ricky> Ah
20:06 < mmcgrath> But once the release was out the door.... Things went great.
20:06 < mmcgrath> Anyone have anything else on this topic?
20:06 < rjune_wrk> why are they behind the balancer, but the slowest box has to face
the cold cruel world all alone?
20:07 < mmcgrath> rjune_wrk: well, we've got 4 sites, only one of which has a
20:07 < mmcgrath> so we've got 5 proxy servers, 2 behind the balancer, the
others are all at the various sites.
20:07 -!- che [n=che@redhat/che] has joined #fedora-meeting
20:07 < rjune_wrk> ah, ok.
20:07 < mmcgrath> Ok, anyone have anything else?
20:07 < mmcgrath> Solid
20:08 < mmcgrath> #topic Infrastructure -- Fedora Community
20:08 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Fedora
20:08 < lmacken> We deployed our first TurboGears2 application this week :)
20:08 < mmcgrath> So fedora community shipped.
20:08 < mmcgrath> lmacken: you want to talk a bit about all of it?
20:08 < lmacken> Sure
20:08 < SmootherFrOgZ> \o/
20:08 < lmacken> Software speaking, not much went wrong.
20:08 * mmcgrath is talking about https://admin.fedoraproject.org/community/
20:08 < lmacken> I hit some insane traceback during staging that was caused by
pointing to the unresolvabl
20:08 < lmacken> e db1.stg, instead of just db1. Other than that, everything went
20:09 < lmacken> It's using both of our new memecached servers, using postgres
on db2 for the feed store, and currently load balancing to mod_wsgi on app1 and 2
20:09 < lmacken> and it's the snappiest I've ever seen the app as of yet
20:09 -!- sseiersen|away is now known as sseiersen
20:09 < lmacken> but anyway, I did another update today to fix a variety of bugs,
and we're going to keep cranking on it
20:09 < mmcgrath> It also means we can, at our leasure, convert our apps to TG2.
20:09 < lmacken> I want to plan a post-mortem when spot gets back from vacation
20:10 < lmacken> so we can talk about what happened, and how we can do it better
20:10 < mmcgrath> lmacken: is the whole tg2 stack in Fedora yet? I need to install
and start playing with it.
20:10 < lmacken> the whole TurboGears2/FedoraCommunity/Moksha stack is in our
fedora-infrastructure repo, and a chunk of it is still under review
20:10 < ricky> Does our tg2 support mean close-to automatic support for pylons as
20:10 < lmacken> ricky: yes
20:10 < ricky> Awesome
20:11 < lmacken> pylons should work fine now
20:11 < mmcgrath> excellent.
20:11 < mmcgrath> Anyone have any questions for Luke on fedora community?
20:11 < lmacken> I will start writing up a TurboGears2 SOP
20:11 < abadger1999> I'd like to get together some questions to ask spot wrt
20:12 < ricky> I guess that can be discussed at the post-mortem as soon as spot is
back - can that be on IRC?
20:12 < abadger1999> That way we can have less round trips to legal.
20:12 < abadger1999> IRC +1
20:12 < mmcgrath> <nod> I just want to be better educated about it before I
copy and paste :)
20:12 < lmacken> yes, it will be public, probably on irc or asterisk
20:13 < rjune_wrk> IRC+1
20:13 -!- ks3 [n=ks3(a)188.8.131.52] has joined #fedora-meeting
20:13 < mmcgrath> Anyone have anything else on this?
20:13 < abadger1999> asterisk -1; IRC is better for public stuff.
20:13 < lmacken> mmcgrath: when do you wanna spread to app3-6?
20:13 < mmcgrath> lmacken: good work, thanks for the time you've put in on it.
20:13 < lmacken> abadger1999: agreed
20:14 < mmcgrath> lmacken: today or tomorrow. I know you did some updates, were
they straight rpm updates?
20:14 < lmacken> mmcgrath: thank you. Thanks for your help as well, couldn't
have done it without you guys
20:14 < lmacken> mmcgrath: yep, straight rpm updates
20:14 < mmcgrath> coolz.
20:14 < mmcgrath> ok so next topic
20:15 < mmcgrath> #topic Infrastructure -- Memcached
20:15 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Memcached
20:15 < mmcgrath> we've been slipping down the slippery slope that is
20:15 < mmcgrath> a month ago we were using it for sessions on the wiki.
20:15 -!- ldimaggi_ [n=ldimaggi(a)pool-96-252-122-35.bstnma.fios.verizon.net] has quit Read
error: 110 (Connection timed out)
20:15 -!- phunster [n=lanny(a)pool-74-105-133-148.nwrknj.fios.verizon.net] has quit
20:15 < mmcgrath> starting today we've got 3 apps using it and dedicated
20:16 < mmcgrath> the fas memcached stuff is going to exist just on the fas servers
for security reasons.
20:16 < mmcgrath> everything else is on memcached[1-2]
20:16 < mmcgrath> ricky: do you want to remove memcached from puppet on app[1-2] or
do you want me to
20:16 < mmcgrath> ?
20:16 < ricky> I'll get that after the meeting
20:16 < mmcgrath> solid
20:16 < mmcgrath> Nothing major here, just mentioning it. Any questions?
20:16 -!- ChitleshGoorah [n=chitlesh@fedora/ChitleshGoorah] has quit Read error: 113 (No
route to host)
20:16 < ricky> Speaking of security, I just want to let everybody know that load
access into memcached = arbitrary command execution with python-memcached
20:17 < ricky> So be careful about that :-)
20:17 < mmcgrath> <nod>
20:17 -!- djf_jeff [n=jeff(a)modemcable026.33-70-69.static.videotron.ca] has quit "I
20:17 < mmcgrath> k
20:18 < mmcgrath> #topic Infrastructure -- fas update
20:18 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- fas update
20:18 < mmcgrath> So we just got done (well, are still doing) the fas updates.
20:18 < mmcgrath> This was a pretty major change as changes go.
20:18 < mmcgrath> Especially in how our shell accounts and aliases are generated.
20:18 < mmcgrath> So keep your eyes out for oddities.
20:19 < mmcgrath> So far things have been going fine, we've been running in
staging for a while now without issue.
20:19 -!- ChitleshGoorah1 [n=chitlesh(a)d54C48C73.access.telenet.be] has quit Connection
20:19 < mmcgrath> I'm going to be moving our fas-sync to run more often on some
of our more public servers
20:19 < mmcgrath> fedorapeople, fedorahosted, cvs, etc.
20:19 < mmcgrath> Any questions there?
20:19 -!- rdieter [n=rdieter@fedora/rdieter] has quit Remote closed the connection
20:19 < ricky> Woohoo :-) My goal is <= 5 minutes sync time
20:20 < mmcgrath> :)
20:20 < mmcgrath> Ok
20:20 < mmcgrath> #topic Infrastructure -- Outage this morning
20:20 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Outage this
20:20 < mmcgrath> So we had some doom this morning for those of you hanging out in
20:20 < mmcgrath> Not sure what happened yet but I opened a bug about it
20:20 < mmcgrath> https://bugzilla.redhat.com/505352
20:20 < mmcgrath> xen13 up and died.
20:20 < mmcgrath> and took db2 with it, twice.
20:20 < mmcgrath> I moved db2 to a different host, hasn't happened since.
20:21 < mmcgrath> at the moment, nothing new on this so I'll move on.
20:21 < mmcgrath> #topic Infrastructure -- Pkgdb update
20:21 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Pkgdb update
20:21 < mmcgrath> abadger1999: we're on for a pkgdb rollout on Monday?
20:21 < abadger1999> that's right.
20:21 < mmcgrath> and this is a wsgi conversion?
20:21 < mmcgrath> any other changes?
20:21 < abadger1999> It's a major update that gets us using wsgi for the
packagedb, all the bugfixes that accrued during change freeze,
20:22 < mmcgrath> I've still got smolt to convert.
20:22 < abadger1999> moving to usernames instead of userids i nthe public API.
20:22 < mmcgrath> after that I *think* all of our tg stuff will be good to go.
20:22 < ricky> Afdter pkgdb and smolt, will all of our apps be on haproxy + wsgi?
20:22 < abadger1999> Switching package status from the orphan owner to the status
20:22 < abadger1999> Lots of stuff changing.
20:23 -!- moixs [n=moixs(a)184.108.40.206] has quit Remote closed the connection
20:23 < lmacken> I'll get a new bodhi ready to go at that time
20:23 < lmacken> I should probably get a new one into staging now so we can test
20:23 < mmcgrath> ricky: I believe so.
20:24 < abadger1999> lmacken: Excellent
20:24 < mmcgrath> lmacken: sounds good.
20:24 < mmcgrath> Anything else on this?
20:24 < abadger1999> Test out admin.stg.fedoraproject.org/pkgdb
20:24 < mmcgrath> hehehe
20:24 < mmcgrath> #topic Infrastructure -- releng1.stg
20:24 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- releng1.stg
20:24 < abadger1999> I'm sure there's more bugs since this has major
20:24 < ricky> After the pkgdb update, let's go through staging
20:24 < mmcgrath> speaking of staging
20:24 < mmcgrath> abadger1999: yeah :)
20:25 < mmcgrath> releng1.stg is nearing rediness.
20:25 < ricky> And make sure to get all the commits we want out of it before
restarting the branch.
20:25 < mmcgrath> lmacken: I need to find out from you and jwb what is missing from
it. I think it's going to take a close eye to make sure we know what we're
putting in staging.
20:25 < mmcgrath> cron jobs and things that might fail, that sort of thing.
20:25 < lmacken> mmcgrath: readonly mount of /mnt/koji?
20:25 < mmcgrath> lmacken: it'll have that.
20:26 < lmacken> ok, to test mashing and whatnot, we'll have to tweak some
20:26 < mmcgrath> lmacken: and I don't think anything mash like is installed
20:26 < mmcgrath> but I figure we'll work through it one boy one until it's
usable for you and jwb.
20:27 < lmacken> ok, cool
20:27 < mmcgrath> I also assume you're going to need some sort of $TEMP_STORAGE
20:27 < lmacken> yeah
20:27 < mmcgrath> that's fairly large. we'll figure that out when the time
20:27 -!- knurd is now known as knurd_afk
20:27 < mmcgrath> was jwb at FAD?
20:27 < lmacken> yep
20:27 < mmcgrath> probably traveling / busy
20:27 < mmcgrath> we'll get ahold of him.
20:27 < mmcgrath> ok, ricky :)
20:27 < mmcgrath> #topic Infrastructure -- Staging
20:27 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Staging
20:28 < mmcgrath> So we need to refresh the staging branch at some point soon.
20:28 < mmcgrath> and make sure all of our staging commits get merged back with
20:28 < mmcgrath> ricky: what are a rundown of some of the commits left?
20:28 -!- cassmodiah [n=cass@fedora/cassmodiah] has quit "Verlassend"
20:28 < mmcgrath> I know pkgdb had a bunch.
20:29 < ricky> Here's the full diff: http://ricky.fedorapeople.org/staging.diff
20:29 -!- phunster [n=lanny(a)pool-74-105-133-148.nwrknj.fios.verizon.net] has joined
20:29 < ricky> It's mostly pkgdb, I thnk
20:29 < ricky> **think
20:29 < ricky> It's not that bad, so we might get away with not blowing it away
20:30 < ricky> But I guess that's something to look at once pkgdb changes have
20:30 < abadger1999> pkgdb will be synced after Monday, so that's good.
20:30 < mmcgrath> Yeah
20:30 < ricky> Cool.
20:30 < abadger1999> Do we have a good recipe for merging the changes yet?
20:30 < ricky> And thanks to mmcgrath for starting to merge often this week
20:30 < mmcgrath> All in all, considering we don't really monitor this
interaction much... things are in better shape then they otherwise could be.
20:30 < mmcgrath> ricky: figured i'd make a habit of it
20:31 < ricky> abadger1999: It'll probably consist mostly of finding out what we
want to keep or throw away
20:32 < ricky> The process of merging isn't that painful at all, just getting
updates on half-finished or abandoned work in staging.
20:32 < abadger1999> ricky: I'm just wondering, can I copy changes from staging
to production, commit, and git will figure it out? Or do I need to use some git command
to pick out specific changesets?
20:33 -!- Sonar_Guy [n=Who@fedora/sonarguy] has joined #fedora-meeting
20:33 < ricky> It's cleaner to cherry-pick, but I *think* hand-copying will work
as well. I haven't tried doing work that way though, so I'm not sure how well it
20:33 < ricky> I think the diff is short enough for us to go through it manually
20:33 < mmcgrath> I've hand-copied with success in the past.
20:33 < mmcgrath> but try to use git just so I'm mo'betta with it.
20:35 < abadger1999> Okay. We'll see what it looks like after Monday's
20:35 < mmcgrath> Ok, anything else on that?
20:35 < mmcgrath> yeah
20:35 -!- Sonar_Guy [n=Who@fedora/sonarguy] has quit Client Quit
20:35 < mmcgrath> With that, we'll open the floor
20:35 -!- Gremlin_ [n=lkrzemin(a)77-97-161-32.cable.ubr04.donc.blueyonder.co.uk] has joined
20:35 < mmcgrath> #topic Infrastructure -- Open Floor
20:35 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Open Floor
20:35 < mmcgrath> Anyone have anything they'd like to discuss?
20:35 < LinuxCode> apart from a million questions I have ?
20:35 < LinuxCode> lol
20:35 < ricky> Ask away :-)
20:36 < LinuxCode> I will have to figure out what is done for what purpose the enxt
20:36 -!- Gremlin_ [n=lkrzemin(a)77-97-161-32.cable.ubr04.donc.blueyonder.co.uk] has quit
20:36 -!- Sonar_Guy [n=Who@fedora/sonarguy] has joined #fedora-meeting
20:36 < LinuxCode> ricky, I ask you guys the next weeks, or we would be here all
20:36 < LinuxCode> lol
20:36 < ricky> feel free to ask in #fedora-admin after the meeting as well
20:36 < LinuxCode> Id like to figure out how everything fits together
20:36 < LinuxCode> sure
20:36 < LinuxCode> I have one question though, the fas update was to kill bugs ?
20:37 < LinuxCode> it said so in the email I had
20:37 < abadger1999> LinuxCode: For some definition of bugs, yes.
20:37 < LinuxCode> abadger1999, I take it to make sure that people participate and
not just sign up then ?
20:37 < abadger1999> Ther were bugs and there was also performance problems that
were causing failures to occur.
20:37 < LinuxCode> ohh ok
20:37 < LinuxCode> i see
20:38 < ricky> The extra @fp.o aliases being created was a bug too
20:38 < LinuxCode> ricky, thats what I meant
20:38 < ricky> Unfortunately, it went unnoticed for a loong time.
20:38 < LinuxCode> would have been nice to give people a week though to sort
20:38 < ricky> The policy hasn't changed, it was just a bug that we weren't
20:38 < LinuxCode> I had the email today saying about the bug
20:39 < LinuxCode> hmm k
20:39 -!- sseiersen is now known as Video_juegos_not
20:39 < LinuxCode> well it doesnt matter now
20:39 < LinuxCode> for me anyway
20:39 < LinuxCode> ;-}
20:39 < mmcgrath> heheh
20:39 < mmcgrath> Anyone have anything else to discuss? If not we'll close the
20:39 -!- Video_juegos_not is now known as sseiersen
20:40 < mmcgrath> Closing in 30
20:40 < mmcgrath> 10
20:40 < mmcgrath> #endmeeting