20:00 < mmcgrath> #startmeeting
20:00 < fedbot> Meeting started Thu Jul 2 20:00:29 2009 UTC. The chair is
mmcgrath.
20:00 < fedbot> Information about MeetBot at
http://wiki.debian.org/MeetBot , Useful
Commands: #action #agreed #halp #info #idea #link #topic.
20:00 < dgilmore> gday mmcgrath
20:00 * ricky
20:00 < mmcgrath> #topic Infrastructure -- Who's here?
20:00 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Who's
here?
20:00 * johe|home takes a seat
20:00 < mmcgrath> dgilmore: how's it going?
20:00 * SmootherFrOgZ is
20:01 * sijis sijis is here.
20:01 * ke4qqq is
20:01 < dgilmore> mmcgrath: 2 builders to go
20:01 < SmootherFrOgZ> dgilmore: for stg ?
20:01 < smooge> hello
20:01 < mmcgrath> dgilmore: excellent, happy to hear it.
20:01 < mmcgrath> Well lets get started
20:01 -!- StabbyMc [n=StabbyMc(a)rrcs-71-41-150-146.sw.biz.rr.com] has left #fedora-meeting
["Stab ya later!"]
20:01 < mmcgrath> #topic Infrastructure -- Tickets
20:01 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Tickets
20:01 < mmcgrath> .tiny
https://fedorahosted.org/fedora-infrastructure/query?status=new&statu...
20:01 < zodbot> mmcgrath:
http://tinyurl.com/47e37y
20:02 < mmcgrath> .ticket 1503
20:02 < mmcgrath> abadger1999: take it
20:02 < zodbot> mmcgrath: #1503 (Licensing Guidelines for apps we write) - Fedora
Infrastructure - Trac -
https://fedorahosted.org/fedora-infrastructure/ticket/1503
20:02 < dgilmore> SmootherFrOgZ: nope
20:02 < abadger1999> So we've had a new license pop up in apps we've written
recently
20:02 < abadger1999> AGPLv3+
20:02 < abadger1999> That's incompatible with GPLv2 which is what the majority
of our apps use presently.
20:03 < abadger1999> After looking over the situation with spot, it seems like it
would be good to move everything to AGPLv3+.
20:03 < dgilmore> im ok with the move
20:03 < abadger1999> (With libraries going to LGPLv2+)
20:03 < smooge> abadger1999, when you say we use.. do you mean we right or other
stuff
20:03 < abadger1999> We write.
20:03 < smooge> s/right/write/
20:03 < smooge> thanks
20:04 < abadger1999> smooge: This would not affect code that we don't write.
20:04 < abadger1999> And it's a recommendation rather than a hard and fast
rule.
20:04 < mmcgrath> abadger1999: have you run into anyone saying "ehh, I
don't think we should do this." ?
20:04 < abadger1999> ie: mdomsch wants mirrormanager to be MIT; mediawiki plugins
should follow mediawiki's license
20:04 < abadger1999> mmcgrath: So far everyone's been positive.
20:04 < mmcgrath> abadger1999: ok, so how do we actually _do_ it?
20:05 < mmcgrath> sed?
20:05 < abadger1999> yeah, we have to replace COPYING files with AGPL/LGPL and then
change the headers in source files.
20:05 < smooge> well you need to look at each app and see if its something we wrote
or pulled in from somewhere else
20:06 < sijis> do you need to get written proof from author before changing?
20:06 < ricky> How urgent is this time-wise?
20:06 < smooge> if its pulled in we need to deal with it.. if its something we wrote
100% we should be able to replace COPYING/headers
20:06 < abadger1999> sijis: for the majority of things no, but I am going to notify
authors of pkgdb and python-fedora before I make chanes.
20:06 < mmcgrath> ricky: I'd say not real urgent, but the longer we wait... the
longer we're going to wait I suspect.
20:06 < ricky> For example, with FAS, I'd like to eventually rewrite the OpenID
provider part instead of dealing with licensing pain because of samadhi or anything.
20:06 < abadger1999> sijis: The CLA gives us the ability to do a relicense if the
contribution was made without an explicit license.
20:07 < mmcgrath> abadger1999: some seemed timid about that on f-a-b. I'm less
timid.
20:07 < abadger1999> <nod> ricky the other option is to find out what jcollie
thinks about AGPLv3+
20:07 < mmcgrath> but we should ask
20:07 < mmcgrath> abadger1999: lets take an app like fas first.
20:07 < mmcgrath> just see how it goes.
20:08 < abadger1999> yeah, it's common courtesy and also gives people a chancce
to holler "Oh wait, I actually didn't own the copyright to that code..
sorry."
20:08 -!- mcepl [n=mcepl(a)49-117-207-85.strcechy.adsl-llu.static.bluetone.cz] has left
#fedora-meeting []
20:08 < mmcgrath> abadger1999: are you going to lead the effort on this?
20:08 < abadger1999> I'd like to do python-fedora soon It's moving to
LGPLv2+ which is more permissive
20:08 < mmcgrath> should we open a ticket for each app?
20:08 < sijis> how many apps are we talking about for this? +/-15?
20:08 < abadger1999> mmcgrath: I can. Yes, each app.
20:08 < mmcgrath> sijis: less then 15
20:08 < abadger1999> sijis: Less htan 15
20:09 < mmcgrath> abadger1999: sounds good, so anything else?
20:09 < abadger1999> A ticket for each app will let us come back next week and say
-- half of our app authors like a licensing policy but don't want to change *their*
app.
20:09 < abadger1999> Which would mean we need to rethink.
20:10 < abadger1999> I think that's all unless someone wants to shout that
it's a bad idea now :-)
20:10 -!- kolesovdv [n=kolesovd(a)82.162.141.18] has joined #fedora-meeting
20:10 < mmcgrath> anyone have anything to say? If not now, take it to the list.
20:10 < mmcgrath> and do it sooner, not later.
20:10 < mmcgrath> Ok, so next topic
20:10 < mmcgrath> #topic Infrastructure -- The merge, outages and issues.
20:10 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- The merge,
outages and issues.
20:11 < mmcgrath> So we had a merge last week.
20:11 < mmcgrath> and since the merge we've had some issues
20:11 < mmcgrath> and it's not something obvious.
20:11 < smooge> define merge for me?
20:11 < mmcgrath> and, in fact, could be completely unrelated.
20:11 < mmcgrath> smooge: merge from staging to master branches in puppet.
20:11 < ricky> smooge: We made a ton of changes in the staging branch and merged
them to production :-)
20:11 < mmcgrath> Which basically involved refactoring a bunch of puppet code,
cleaning things up, creating some new modules, etc, etc.
20:11 < mmcgrath> I've not seen a wiki outage since yesterday.
20:12 < mmcgrath> I need to go through the logs and look.
20:12 -!- cassmodiah [n=cass@fedora/cassmodiah] has quit Remote closed the connection
20:12 < mmcgrath> while doing some digging we, just in general, found strange issues
in our environment.
20:13 < smooge> mmcgrath, ricky thanks..
20:13 < smooge> what have been the strange ones
20:13 < mmcgrath> for example -
http://mmcgrath.fedorapeople.org/proxy-errors.html
20:13 < mmcgrath> 200,000+ 502's per day.
20:13 < mmcgrath> just seems massive to me.
20:14 < ricky> In terms of the big outages, they've all seemed to happen during
mysql database backups (which lock tables) or smolt render stats jobs.
20:14 < ricky> The proxy errors and 500s seem to be something else though.
20:14 < mmcgrath> <nod>
20:14 < mmcgrath> and our current lead on the 500's errors for fas is a new
mod_wsgi
20:14 < ricky> Have the 500 errors stayed normal?
20:14 < mmcgrath> jbowes is working on that.
20:15 < ricky> (As in, have they gone up after the merge or not?)
20:15 < mmcgrath> ricky: hard to say
20:15 < mmcgrath>
http://mmcgrath.fedorapeople.org/JuneErrors.html
20:15 < mmcgrath> I'll re-check today now that it's been a few more days.
20:15 < mmcgrath> clearly we had a major spike
20:16 < sijis> mmcgrath: the first graph shows it being mostly proxy2
20:16 < mmcgrath> but it seems to have gone back down.
20:16 < ricky> Strange.
20:16 < mmcgrath> sijis: yeah, and proxy2 is an odd beast.
20:16 < mmcgrath> proxy2 is load balanced with proxy1 behind the PHX balancer.
20:16 < mmcgrath> _however_
20:16 < mmcgrath> anything in phx uses proxy2 directly to get to the account
system.
20:16 < mmcgrath> which not only includes shell accounts.
20:17 < mmcgrath> but also includes our web applications contacting fas for session,
auth, etc.
20:17 < mmcgrath> which is a significant amount of traffic.
20:17 < smooge> interesting.. is there a reason for just proxy2?
20:17 < ricky> Funny that proxy1 seems fine.
20:17 < mmcgrath> ricky: well it does get a lot less traffic.
20:17 < ricky> Like it didn't jump significantly at all.
20:17 < ricky> I guess.
20:17 < mmcgrath> smooge: the network team won't let us contact the balancer IP
directly.
20:18 < sijis> so you are forced to pick a proxy?
20:18 < smooge> ah ok could we setup another proxy?
20:18 < mmcgrath> smooge: we have two of them there.
20:18 < mmcgrath> but no good way to balance between the two of them.
20:18 < mmcgrath> we could put a load balancer in there, but it'd be just
another box, and would need to be rebooted as often as proxy2 is anyway
20:18 < ricky> Is the problem really coming from our PHX admin.fp.o setup though?
20:19 < smooge> mmcgrath, no what I meant was one that was just for that so we could
cut down on what might be causing the erorrs?
20:19 < ricky> The 502s really jumped everywhere, so that's what I want to know
the root cause of.
20:20 < smooge> so if its a bruteforce attack on stuff we could get an idea of what
app is being targeted or soemthing
20:20 < mmcgrath> I think the errors are on our end, I need to do more log checking
to know for sure though
20:20 < ricky> But the brute force shouldn't be causing 502, it should be
working :-)
20:20 < mmcgrath> but yeah we can add and remove more proxy servers in PHX if we
want to
20:20 -!- JSchmitt [n=s4504kr@fedora/JSchmitt] has quit Read error: 104 (Connection reset
by peer)
20:20 < ricky> mmcgrath: Can we separate that graph into apache 502s and haproxy
502s?
20:21 < ricky> Right now they're lumped together in the source where you're
getting it from, right?
20:21 -!- ddumas [n=ddumas(a)h69-131-97-205.wltonh.dsl.dynamic.tds.net] has joined
#fedora-meeting
20:21 < mmcgrath> ricky: I don't think so, because if haproxy or the app server
returned a 502, apache would log a 502.
20:21 < mmcgrath> so proxyX will always have our largest number of 502's
20:21 < mmcgrath> then haproxy (if we're logging that, not even sure)
20:21 < mmcgrath> then the app server
20:22 < mmcgrath> although the app servers probably don't throw 502
20:22 < ricky> mmcgrath: But some 502s are coming from apache, as in proxy1
couldn't contact locahost:10009
20:22 < ricky> Those are the strangest ones to me.
20:22 < mmcgrath> I'll have to look closer then.
20:22 < sijis> firewall?
20:23 < ricky> sijis: I don't think so - it definitely works a large percent of
the time
20:23 < mmcgrath> sijis: I'd actually think that's the app server not
responding to haproxy, and thus not responding to the proxy server.
20:23 < ricky> But that should strictly cause haproxy 502s not apache 502s,
correct?
20:23 < mmcgrath> and I'm not seeing us hitting our haproxy limit.
20:23 < ricky> and we've seen both :-(
20:23 < mmcgrath> ricky: when looking at the logs, how can you tell the difference?
20:24 < mmcgrath> oh from it saying it couldn't contact localhost:10009
20:24 < ricky> I'm not sure. I'd expect the apache 502s to show up in the
apache error log and both types of 502s to show up in the error log.
20:24 < ricky> I'll have to verify that tohugh.
20:24 < ricky> **though
20:24 < mmcgrath> hm
20:24 < mmcgrath> hm
20:24 < mmcgrath> hmmmm
20:24 < ricky> Was your source for these graphs the error log or the access log?
20:25 < mmcgrath> acciess I believe
20:25 < sijis> is haproxy on a different server or on proxy2?
20:25 * mmcgrath looks
20:25 < mmcgrath> sijis: each proxy server has it's own haproxy service on the
same host
20:25 < mmcgrath> ricky: access.log
20:26 < mmcgrath> perhaps we should continue discussing this after the meeting.
20:26 < ricky> Ah, OK.
20:26 < mmcgrath> any objections?
20:26 < ricky> Sure thing
20:26 < sijis> nope.
20:27 < mmcgrath> # topic Infrastructure -- Eye in know db. - INNODB
20:27 -!- kolesovdv [n=kolesovd(a)82.162.141.18] has quit Remote closed the connection
20:27 < mmcgrath> #topic Infrastructure -- Eye in know db. - INNODB
20:27 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Eye in know
db. - INNODB
20:27 < mmcgrath> ricky: this one's you. Talk about your plans, what's
going on, what's going wrong, etc.
20:27 < smooge> is that a rock band?
20:27 < ricky> Any MySQL experts around, by the way? :-)
20:27 < mmcgrath> ricky: abadger1999 is a mysql expert
20:27 < Jeff_S> ricky: for some definition of expert
20:27 < mmcgrath> :-P
20:28 < ricky> Part of the big outages we've seen since the merge seems to be
due to mysql backups (and smolt's stats refresh script, which might be a separate
problem)
20:28 < ricky> We've seen this behavior with the zabbix database, where the
backup would lock entire tables
20:28 < abadger1999> ricky: Yep, of the yum erase '*ysql' ; yum install
'postgres*' variety
20:28 < ricky> abadger1999: Hehe
20:28 * mmcgrath notes we've always had a small problem with backups and outages.
But they've been tiny blips. Lately they've been throwing nagios alerts.
20:29 < smooge> how many mysql databases do we have?
20:29 < ricky> We'd like to move to using the --single-transaction option to
mysqldump, which combined with InnoDB, should make backups not lock the entire table
20:30 < Jeff_S> ricky: yes!
20:30 < ricky> THe main mysql usage we have is mediawiki, smolt, and zabbix
20:30 < ricky> Although we have a few others for stuff like cacti, prelude/prewikka,
etc.
20:30 < Jeff_S> ricky: FWIW, we've also had good luck with
http://www.zmanda.com/backup-mysql.html (community edition)
20:30 < smooge> ricky, are they seperate servers or one single one
20:30 -!- kolesovdv [n=kolesovd(a)82.162.141.18] has joined #fedora-meeting
20:30 < ricky> Jeff_S: Thanks, I'll take a look at that later
20:30 < ricky> smooge: They're all on db1
20:30 < mmcgrath> smooge: all mysql db's are on db1
20:31 < ricky> So far, the biggest pain we've had so far is the host_links table
in smolt
20:31 < mmcgrath> ricky: and how big is it?
20:31 < mmcgrath> O:-)
20:31 < ricky> It has above 70M rows, and I haven't gotten a single successful
conversion to InnoDB yet.
20:31 < ricky> And the thing with --single-transaction is that the tables need to be
InnoDB to be sure that everything gets dumped in a consistent state
20:31 < Jeff_S> but single-transaction will probably solve your main problem of
locking the table(s)
20:32 < abadger1999> ricky: We're able to dump that table? Are we able to
reload it except as innodb?
20:32 < smooge> wow thats quite a bit
20:32 < mmcgrath> ricky: and what are the downsides to innodb? (space, etc, etc)
20:32 < abadger1999> slower
20:32 < Jeff_S> mmcgrath: slower at certain operations
20:32 < ricky> So the approaches that we've tried so far are: converting using
alter table, and sedding a dump to change the table type, and loading it.
20:32 < mmcgrath> how much slower?
20:32 < ricky> The first didn't finish after some large number of hours, and the
second is going now.
20:32 < ricky> mmcgrath: I'm actually not that sure about the downsides yet.
Apparently loading huge tables is a huge pain.
20:33 < mmcgrath> ricky: I'm going to want render-stats metrics too
20:33 < Jeff_S> mmcgrath: depends on the dataset & queries. the locking though
more than makes up for it IMO
20:33 < ricky> Also, some tables needed MyISAM for full text search - the only table
affected by this is mediawiki's searchindex tables
20:33 < abadger1999> :-(
20:33 < ricky> (Which is just a copy of another InnoDB table, I believe)
20:33 < mmcgrath> ricky: and, in theory, we'll be able to get rid of that when
we have a fedora search engine.
20:33 < ricky> Hopefully.
20:34 < ricky> Anyway, we'll probably have a mysql outage some time in the
future once we get a successful test in staging.
20:34 < Jeff_S> mmcgrath: one of our past employees wrote this, I think it explains
the reasons for using InnoDB pretty well
http://tag1consulting.com/MySQL_Engines_MyISAM_vs_InnoDB
20:34 < mmcgrath> ricky: yeah, how have the other conversions gone?
20:34 < ricky> what might be the case now is that maybe our configs aren't tuned
for large innodb tables.
20:34 < smooge> ok what books/sites should I read to catch up how to help this.
(DB's are not my specialty :/)
20:35 -!- hanthana [n=hanthana(a)124.43.57.16] has quit Remote closed the connection
20:35 < ricky> mmcgrath: All of the other tables in the smolt db other than
host_links have finished in <20 minutes
20:35 < ricky> Apart from the smolt db, most of the mediawiki db is already innodb
20:36 < ricky> The other databases that need conversions are: cacti,
prelude._format, prewikka, and transifex (which isn't used anymore anyway)
20:36 < mmcgrath> ricky: I believe I went through and did some innodb conversions
back in the day on some of those.
20:36 -!- openpercept_ [n=openperc@fedora/openpercept] has joined #fedora-meeting
20:36 -!- tatica is now known as tatica-out
20:36 -!- sharkcz [n=dan(a)plz1-v-4-17.static.adsl.vol.cz] has quit "Ukončuji"
20:36 < ricky> prelude and prewikka are pretty much dispensable since that stuff is
still being tested (lmacken even purged and recreated some of those dbs recently)
20:37 < mmcgrath> ricky: how big were those dumps?
20:37 -!- openpercept [n=openperc@fedora/openpercept] has quit Nick collision from
services.
20:37 < ricky> So smolt is basically the big hurdle - although I have some questoins
about the smolt upgrade and the db changes there
20:37 < ricky> The dump of the smolt database is 2.5G
20:37 * lmacken looks at the time, and rolls in late
20:37 < mmcgrath> ricky:
20:38 < mmcgrath> alter table host modify column cpu_model varchar(80);
20:38 < mmcgrath> alter table host add column cpu_stepping int(11) DEFAULT NULL;
20:38 < mmcgrath> alter table host add column cpu_family int(11) DEFAULT NULL;
20:38 < mmcgrath> alter table host add column cpu_model_num int(11) DEFAULT NULL;
20:38 < mmcgrath> that's the smolt upgrade.
20:38 < ricky> mmcgrath: Oh, OK - that's no problem at all then.
20:38 < ricky> The host table took <20 minutes, so we can do that before or
after, and it's fine
20:38 * mmcgrath doesn't really even know what "int(11)" means
20:38 < lmacken> have you guys been using SQLAlchemy-migrate for that stuff? or
doing it by hand?
20:38 < mmcgrath> I need to look that up :)
20:39 < mmcgrath> lmacken: honestly I can't stand alchemy-migrate so I've
been doing it by hand.
20:39 < lmacken> mmcgrath: heh. I've never used it before
20:39 < mmcgrath> :)
20:39 < mmcgrath> ricky: ok, so anything else on the db front?
20:40 < ricky> Nope, but if anybody knows a lot about MySQL, let us know about your
experiences with stuff like this
20:40 < ricky> Jeff_S: Thanks again for the links!
20:40 < mmcgrath> k
20:40 < Jeff_S> ricky: np. I'm glad to have our current DBA lend a hand if
needed
20:40 < mmcgrath> #topic Infrastructure -- Posse
20:40 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Posse
20:41 < mmcgrath> So I haven't been as transparent with this as I should be
20:41 < mmcgrath> It's basically this
20:41 < mmcgrath> #link
http://teachingopensource.org/index.php/POSSE_2009
20:41 < mmcgrath> we're providing some guests for a week for them to use.
20:41 < mmcgrath> +1 to open source :)
20:41 < ricky> Is it going to be on fasClient? :-)
20:41 < mmcgrath> ricky: nope, they're completely disconnected atm.
20:42 < mmcgrath> this is their first time through this.
20:42 < ricky> Ah, OK
20:42 < mmcgrath> maybe next year.
20:42 < mmcgrath> but all of these guests are on cnode1
20:42 -!- opossum1er [n=opossum1(a)cnv94-4-88-160-98-200.fbx.proxad.net] has joined
#fedora-meeting
20:42 < mmcgrath> part of the cloud stuff.
20:42 < smooge> what servers are their guest on
20:42 < ricky> Hehe
20:42 < smooge> ah
20:42 < mmcgrath> I ended up not using osuosl1
20:42 < mmcgrath> since it's RHEL5 and for some reason xen+fedora 11 seems to be
my white whale.
20:42 -!- opossum1er [n=opossum1(a)cnv94-4-88-160-98-200.fbx.proxad.net] has quit Client
Quit
20:42 < mmcgrath> but cnode1 was F10, and using KVM worked just fine
20:43 < mmcgrath> Anyone have any other questions on that?
20:43 -!- Pikachu_2014 [n=Pikachu_(a)85-169-128-251.rev.numericable.fr] has quit Read error:
60 (Operation timed out)
20:43 -!- Pikachu_2014 [n=Pikachu_(a)85-169-128-251.rev.numericable.fr] has joined
#fedora-meeting
20:43 < mmcgrath> Ok
20:43 < mmcgrath> #topic Infrastructure -- Open Floor
20:43 -!- fedbot changed the topic of #fedora-meeting to: Infrastructure -- Open Floor
20:43 < mmcgrath> anyone have anything they'd like to discuss?
20:44 < lmacken> I'm going to be deploying a new version of bodhi
tonight/tomorrow to support EPEL :)
20:44 < lmacken> hopefully we'll be able to start queueing updates up tonight
20:44 < smooge> yeah
20:44 < lmacken> and ideally mashing repos tomorrow
20:45 < mmcgrath> lmacken: sounds good
20:45 < mmcgrath> and on a related note, I need to rebuild relepel1
20:45 * mmcgrath fail built it
20:45 < mmcgrath> anyone have anything else?
20:45 < mmcgrath> smooge: ?
20:46 < smooge> sorry
20:46 < smooge> keyboard problems
20:46 < smooge> I am checking to see what boxes need updates and I am working on
seeing what ones I can do
20:46 < smooge> I should have that done by tonight/tomorrow.
20:47 < smooge> After that I am checking to see that func and puppet are working on
the boxes
20:47 < smooge> and then finding out all the secret handshakes and such
20:47 < mmcgrath> heheh
20:47 < mmcgrath> fun times
20:47 < smooge> I should have the func done by friday and then it will be time to
work on zabbix
20:48 < mmcgrath> smooge: excellent.
20:48 < mmcgrath> Ok, and with that if no one has anything else we'll close in
30
20:48 < smooge> zabbix will be next weeks project
20:48 < smooge> done
20:49 < mmcgrath> ok everyone, thanks for coming!
20:49 < mmcgrath> #endmeeting
20:49 -!- fedbot changed the topic of #fedora-meeting to: Channel is used by various
Fedora groups and committees for their regular meetings | Note that meetings often get
logged | For questions about using Fedora please ask in #fedora | See
http://fedoraproject.org/wiki/Meeting_channel for meeting schedule
20:49 < fedbot> Meeting ended Thu Jul 2 20:49:12 2009 UTC. Information about
MeetBot at
http://wiki.debian.org/MeetBot .
20:49 < fedbot> Minutes:
http://www.scrye.com/~kevin/fedora/fedora-meeting/2009/fedora-meeting.200...
20:49 < fedbot> Log:
http://www.scrye.com/~kevin/fedora/fedora-meeting/2009/fedora-meeting.200...