#fedora-meeting: Infrastructure (2014-02-13)
Meeting started by nirik at 19:00:46 UTC. The full logs are available at
* welcome y'all (nirik, 19:00:47)
* New folks introductions and Apprentice tasks. (nirik, 19:02:36)
* Applications status / discussion (nirik, 19:04:36)
* FedOAuth 2.0 is currently in staging, please test it and report
serious issues! (puiterwijk, 19:04:58)
* bodhi 0.9.8 to prod this week,
lots of bugfixes (nirik, 19:05:37)
* plans are afoot for us to get bugzilla fedmsg's sometime in the next
3-4months. This is really good news! :) (nirik, 19:06:21)
* new datagrepper deployed earlier this week with "html cards"
enhancements from Charul (threebean, 19:08:02)
* LINK: https://apps.fedoraproject.org/datagrepper
* if no issues are found, FedOAuth 2.0 will be promoted to production
on saturday (puiterwijk, 19:08:13)
* new fedora-packages deployed earlier this with, integrating that
datagrepper feature (threebean, 19:08:24)
* LINK: https://apps.fedoraproject.org/packages
* LINK: https://github.com/Debian/dcs
* Sysadmin status / discussion (nirik, 19:26:15)
* smooge on site until friday (nirik, 19:27:01)
* unbound instances moved to ansible (nirik, 19:27:13)
* gluster module ported to an ansible role (threebean, 19:27:58)
* updates being applied to all instances (but no reboots needed)
* Upcoming Tasks/Items (nirik, 19:36:36)
* LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
* Open Floor (nirik, 19:55:28)
Meeting ended at 19:59:22 UTC.
Action Items, by person
People Present (lines said)
* nirik (122)
* threebean (40)
* pingou (36)
* mirek (22)
* willo (18)
* puiterwijk (14)
* lmacken (9)
* adimania (7)
* zodbot (4)
* tflink (4)
* dgilmore (4)
* abadger1999 (3)
* webpigeon (3)
* relrod (3)
* janeznemanic (2)
* mariocav (2)
* masta (1)
* smooge (0)
* mdomsch (0)
19:00:46 <nirik> #startmeeting Infrastructure (2014-02-13)
19:00:46 <zodbot> Meeting started Thu Feb 13 19:00:46 2014 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
19:00:46 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:47 <nirik> #meetingname infrastructure
19:00:47 <nirik> #topic welcome y'all
19:00:47 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch
threebean pingou puiterwijk
19:00:47 <zodbot> The meeting name has been set to 'infrastructure'
19:00:47 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou
puiterwijk relrod smooge threebean
19:00:52 * relrod here!
19:00:54 * mariocav is here
19:01:04 * threebean is here
19:01:08 * mirek is here
19:01:30 <lmacken> me
19:01:38 * willo present
19:01:49 * webpigeon is here
19:01:52 * adimania is here
19:02:13 * puiterwijk is here
19:02:30 <nirik> cool. :) welcome everyone.
19:02:36 <nirik> #topic New folks introductions and Apprentice tasks.
19:02:44 <nirik> any new folks like to introduce themselves?
19:02:52 <nirik> or apprentices with questions or comments or ideas?
19:03:42 <janeznemanic> hi i'm present too
19:04:28 <nirik> ok
19:04:36 <nirik> #topic Applications status / discussion
19:04:50 <nirik> any application news or plans or discussion.
19:04:58 <puiterwijk> #info FedOAuth 2.0 is currently in staging, please test it and
report serious issues!
19:05:00 <threebean> all kinds of good stuff happening. maybe too much. I
can't keep track of it ;)
19:05:18 <nirik> threebean: yeah, I was just going to mention a few, and
couldn't decide what to mention. ;)
19:05:19 <lmacken> I pushed out bodhi 0.9.8 to prod this week,
lots of bugfixes
19:05:34 <lmacken> on to 2.0 now :)
19:05:37 <nirik> #info bodhi 0.9.8 to prod this week,
lots of bugfixes
19:05:47 <nirik> lmacken: cool. ;)
19:05:55 <threebean> lmacken: awesome. that opens the door to a couple new badges
that are in the queue :)
19:06:07 <lmacken> threebean: yep, those should be good to go now.
19:06:21 <nirik> #info plans are afoot for us to get bugzilla fedmsg's sometime
in the next 3-4months. This is really good news! :)
19:06:29 <threebean> \ó/
19:06:31 <webpigeon> puiterwijk: linky?
19:06:53 <puiterwijk> webpigeon: id.stg.fedoraproject.org
19:07:01 <webpigeon> thanks
19:07:07 <mirek> I have some updates for Copr in queue, but since I will be on
holidays next week I will postpone deployment till i return
19:07:34 <nirik> mirek: sounds good. You saw that idea to have some sort of
fedorahosted -> copr integration?
19:07:35 <tflink> puiterwijk: does that include persona provider support?
19:07:40 <puiterwijk> tflink: yes, it does
19:07:54 <puiterwijk> oh
19:08:02 <threebean> #info new datagrepper deployed earlier this week with
"html cards" enhancements from Charul
19:08:07 <nirik> mirek: also, we can look at backups... we can have our backup
server reach out to the cloud and backup stuff for copr.
19:08:11 <threebean> https://apps.fedoraproject.org/datagrepper
19:08:13 <puiterwijk> #info if no issues are found, FedOAuth 2.0 will be promoted to
production on saturday
19:08:20 <mirek> nirik: not sure which one you think, there was a lot of ideas on
devconf and around
19:08:24 <threebean> #info new fedora-packages deployed earlier this with,
integrating that datagrepper feature
19:08:31 <threebean> https://apps.fedoraproject.org/packages
19:08:33 * pingou late
19:08:42 <mirek> nirik: I'm backing up copr into OS1 right now
19:09:01 <nirik> mirek: ah ha. cool. so, if need be you could fire it up there?
19:09:19 <pingou> fedocal 0.4.6 is in stg since today (0.4.5 was in since last
19:09:28 <mirek> nirik: but just rpms, because that is plain disk, db is on normal
storage (i.e. I hope that is at least two disk in mirror)
19:09:44 <lmacken> threebean: oh excellent, I didn't know that went live
already. So good.
19:09:47 * dgilmore arrives
19:09:50 <mirek> nirik: err, I'm backing up even db
19:09:53 <nirik> mirek: could do a db dump and back that up.
19:09:56 <nirik> ok, cool.
19:10:05 <mirek> yes, db dump
19:10:34 * nirik wonders if we shouldn't have some kind of timeline or list for
things... we have so many apps in the air now.
19:10:56 <mirek> and yes, we have copr probe for nagios based on fedmsg.
19:10:56 <pingou> we do indeed
19:11:18 <pingou> mirek: which timeframe did you set finally?
19:11:32 <nirik> mirek: yeah, we should probibly bump up the time some. we did bump
it to 3 hours, but even thats low.
19:11:56 <nirik> perhaps 6 ?
19:11:57 <mirek> i'm right now preparing really small package, which I plan to
build every hour
19:12:07 <threebean> oo
19:12:09 <mirek> so every hour should come notif
19:12:18 <pingou> that's one option :)
19:12:24 <nirik> hum...
19:12:29 <nirik> I was wondering about that...
19:12:47 <nirik> could we get fedmsg to do pings? some way to submit a message thats
just a 'I am alive and processing ok'/
19:12:56 <nirik> or could we never be sure it really hits all the messages it
19:13:12 <threebean> pingou and I talked about that some over the weekend..
19:13:26 <abadger1999> nirik: Like haproxy, does, you mean?
19:13:31 <mirek> well sometimes backend is alive, but workers are stuck, so you are
not sure unless you really try to build package
19:13:48 <mirek> small package, which already has been build should take just 45
19:14:10 <nirik> abadger1999: sort of, but more application side... it emits all the
messages it can/normally does, but with a 'test' or 'ping' set so it
doesn't show up in the regular stream
19:14:16 <threebean> nirik: the short answer is no. I'd be glad to talk about
the longer answer later if anyone would like..
19:14:32 <nirik> fair enough. Just a thought, I trust your folks judgement.
19:15:32 <nirik> any other application side news?
19:15:38 <pingou> summershum? :)
19:15:45 <nirik> pingou: where were we on pkgdb2? waiting on new fedoauth?
19:15:47 <nirik> ah, that too.
19:16:05 <pingou> nirik: pkgdb2 waits on pkgdb-cli that waits on python-fedora that
waits on fedoauth
19:16:26 <nirik> nice trail. ;)
19:16:27 <nirik> ok
19:16:29 <pingou> yup :)
19:17:01 <pingou> the svg on thisweekinfedora should be a little more readable next
19:17:19 <pingou> (can hardly do worst atm)
19:17:43 <pingou> otherwise, for summershum, what do you guys think?
19:17:55 <nirik> could be nice. ;)
19:18:11 <mirek> what is summershum?
19:18:11 <puiterwijk> pingou: well, you don't need to wait for FedOAuth anymore,
19:18:30 <puiterwijk> at least for dev purposes
19:18:35 <nirik> mirek: a planned app to index/checksum sources.
19:18:58 <nirik> mirek: so, we could then query for 'how many copies of COPYING
are in what packages' or 'how many have this md5.c file'
19:19:00 <pingou> mirek: indexes the sha1sum of every file on every package
19:19:18 <mirek> aha thanks
19:19:28 <pingou> I thought building the API for this would make a good GSoC
19:19:45 <pingou> well, s/I/Ralph and I/
19:20:00 <nirik> (well, if we are in GSoC this year)
19:20:04 <pingou> true
19:20:31 <willo> sounds so awesome. great stats to come from that
19:20:48 <abadger1999> might want to see if you can get a little dmalcolm time to
ask about that as well -- I think he worked on something similar at one point.
19:20:49 <threebean> willo: oh, you were asking about busmon last week?
19:20:51 <nirik> it might be nice to have something like that debian source searcher
19:20:58 <nirik> I don't know if that could be on top of this or not.
19:21:01 <threebean> willo: I have a rewrite in progress I'd be glad to share
with you if you'd like to hack on it
19:21:23 <willo> threebean: yeah, cool. would be great
19:21:25 <pingou> nirik: sounds doable
19:21:51 <nirik> https://github.com/Debian/dcs
19:22:41 <nirik> I can see several uses for something like that...
19:22:54 <nirik> "find out of any other packages are using this thing I just
19:22:55 <pingou> nirik: this is searching within the files
19:22:59 <nirik> yeah.
19:23:06 <nirik> so, different/harder
19:23:12 <threebean> willo: cool. I'll push it on a branch and let you know
19:23:24 <pingou> which is something that was requested on the list, but imho a
little different from the current idea of summershum
19:23:33 <willo> threebean: great
19:23:40 <willo> pingou: shummershum-ng
19:23:41 <nirik> right, it might be seperate, or related, or on top, not sure.
19:23:42 <willo> :)
19:23:46 <pingou> willo: \ó/
19:24:08 <nirik> anyhow, just something to think on...
19:24:30 <nirik> even just a search of spec files/patches would be interesting...
"how many packages use BuildRoot:" or whatever
19:24:51 <pingou> yeah that sounds quite nice
19:25:26 <nirik> anyhow, any other applications news ?
19:26:15 <nirik> #topic Sysadmin status / discussion
19:26:23 <nirik> so, on the sysadmin side of things...
19:26:35 <nirik> smooge is out at our datacenter doing bunches of hands on things.
19:27:01 <nirik> #info smooge on site until friday
19:27:13 <nirik> #info unbound instances moved to ansible
19:27:15 <mirek> are the ARMs in fedora cloud network already?
19:27:25 <nirik> mirek: not yet, but very soon. ;)
19:27:27 <nirik> on that...
19:27:39 <nirik> we are going to move one of our arm chassis over to the cloud
19:27:45 <nirik> thats 24 SOCs.
19:27:58 <threebean> #info gluster module ported to an ansible role
19:28:03 <nirik> we can then use some of them for copr and some for qa/packagers
19:28:31 <nirik> mirek: we need to decide if we want copr to install them clean each
build, or just use existing installs.
19:28:52 <nirik> threebean: yes! Thank you. Although it needs a bit of tweaking to
be idempotent it seems like...
19:29:11 <mirek> I will have use for 8 of them (right now), rest of them can be used
for other purposes
19:29:13 <adimania> I guess unbound is ported fully as well.
19:29:31 <mirek> nirik: I would rather see reinstall before each build
19:29:32 <nirik> mirek: sounds good. I can let you know once we have it setup.
19:29:55 <nirik> mirek: ok, that just would involve ipmitool to set it to pxeboot,
power on, wait for install to finish and use.
19:30:05 <dgilmore> nirik: we can script rebuilds using ipmitool
19:30:06 * mirek nod
19:30:06 <nirik> adimania: yep. ;) Thanks for that work.
19:30:15 <nirik> dgilmore: yep. ;) that was my plan.
19:30:17 <adimania> :)
19:30:25 <threebean> nirik: cool, I'll have a look at it again when I get around
to tagger/packages -> ansible.
19:30:43 <pingou> we added --extra-vars="testing=True" to the
updates_packages and sign_and_import playbooks
19:30:44 <nirik> janeznemanic: I still need to look at your bodhi and smtp-mm
playbook work. ;) it's on my list... thanks for working on it.
19:30:50 <dgilmore> nirik: mirek came and saw me today and I let him know its
19:30:57 <adimania> i can pick up another module. Any easyfixes?
19:30:58 <nirik> excellent.
19:31:22 <nirik> adimania: I was just going to file one on paste/sticky-notes. will
do so when I get a chance.
19:31:34 <janeznemanic> nirik: no problem
19:31:44 <adimania> nirik, great. I'll pick it up.
19:31:59 <nirik> also, I was going to file another easyfix for someone to check our
phx2 dns... see what hosts are not responding to ping (and likely gone) so we can recover
19:32:18 <lmacken> pingou: nice, yeah I used the testing var yesterday. Real handy
19:32:44 <pingou> I had a problem with the update_packages today, the yum clean all
19:32:49 <nirik> finally, I have been updating machines (but only rebooted stg). I
am going to continue that today and get everything updated... should not hopefully have
any outage causing issues.
19:33:21 <nirik> #info updates being applied to all instances (but no reboots
19:34:13 <nirik> On our netapp slowness... there was work tuesday night to try and
upgrade it. It failed to work. There is supposed to be a new plan from netapp folks later
today and likely another attempt and updating. :(
19:35:21 <nirik> lets see....
19:35:54 <nirik> I'm sure there's more stuff, but not sure what. ;)
19:36:36 <nirik> #topic Upcoming Tasks/Items
19:36:36 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
19:36:47 <nirik> anything coming up folks would like to note or schedule.
19:37:10 <threebean> We still have no ETA on a F21 release cycle?
19:37:17 <mirek> I'm next week on holidays
19:37:30 <pingou> enjoy :)
19:37:32 <threebean> mirek: great; enjoy :)
19:37:46 <nirik> threebean: aug/sept... but nothing definite.
19:38:07 <nirik> Hopefully things will come more into focus in the next few weeks as
working groups submit what they want to actually deliver
19:38:19 <nirik> mirek: going somewhere fun I hope?
19:38:45 <mirek> skiing with kids - if there will be some snow
19:38:48 <willo> do we have an infrastructure freeze associated with release?
19:38:57 <nirik> mirek: fun. ;)
19:39:09 <willo> if so, how long before?
19:39:10 <mirek> we have very hot winter this year :(
19:39:12 <nirik> willo: yeah, we do freezes 2 weeks before each release
19:39:35 <nirik> mirek: I think folks in the south of the us would happily send you
some snow. ;)
19:39:54 <mirek> I would be glad to exchange :)
19:39:57 <willo> nirik: great
19:40:13 <threebean> oh, some of us have been discussing the potential for FADs in
the coming few months, but nothing serious or focused yet.
19:40:29 <nirik> willo:
has some more
19:40:46 <nirik> threebean: yeah, I would really like to do one. :)
19:40:48 <puiterwijk> threebean: I propose a FAD for moving everything to OpenID
once FedOAuth 2.5 hits prod
19:40:49 <pingou> there would be budget on OSAS to make an infra FAD
19:40:52 <threebean> some ideas of things to get together one: FAS3, bodhi2,
19:41:23 <nirik> ansible migrating, openid switching
19:41:30 * threebean nods
19:41:32 <threebean> yes
19:42:08 <threebean> some of the final ansible pieces will be tricky (like the proxy
stuff) and would definitely benefit from multiple people being present
19:42:11 <nirik> just for the 'lets actually get it done' I am liking the
bodhi2 idea. ;)
19:42:25 <nirik> yeah.
19:42:30 <threebean> yeah, well, fedora.next I think will put some new ideas into
the mix for bodhi2
19:42:42 <nirik> it could well do so, yep.
19:42:44 <threebean> the more complicated karma system, and more feedback from
taskotron to gate updates?
19:43:01 <pingou> I'm thinking maybe two groups
19:43:01 <threebean> anyways, that's just more reason to get together to
brainstorm and implement
19:43:09 <pingou> FAS3 in one room, Bodhi2 in the other?
19:43:11 <nirik> theres also talk of a 'less constrained' repo of packages,
possibly bodhi would need to be used to mash/manage that too...
19:43:20 <threebean> right, right
19:43:43 <nirik> and scls, etc
19:43:53 <nirik> (alhough they may just be rpms like any other)
19:44:39 <nirik> pingou: we could try multiple goals, but fads seem to do best when
there's one specific goal thats reachable...
19:44:53 <nirik> if people get pulled from a to work on b and back they might not
finish a or b
19:45:38 <nirik> but some things might also be very related.
19:45:49 * threebean nods
19:45:54 <nirik> if bodhi2 and taskotron are interacting, it might be good to work
on both at the same fad.
19:46:09 <nirik> and... tflink and lmacken are both here in colorado, so if we did
one in denver...
19:46:11 <puiterwijk> like openid intersects with about every other app we come up
19:46:18 <threebean> nirik: heh, I was just about to say as much.
19:46:41 <threebean> puiterwijk: a lot of openid interactions we can
implement/deploy on a case by case basis though, no?
19:46:57 <pingou> nirik: we need to figure out wrt costs, but Denver is clearly a
sensible place ;-)
19:47:01 <dgilmore> nirik: i can get a direct flight to Denver
19:47:02 <threebean> as in, once FedOAuth 2.5 is deployed, we can update each
application one after another over time.
19:47:14 <puiterwijk> threebean: yep. but I think we should try to move that
19:47:15 <nirik> yeah.
19:47:23 <threebean> puiterwijk: agreed.
19:47:48 <puiterwijk> threebean: especially when the new OpenID specs land, that
should give us a great reason to push forward more
19:47:49 <lmacken> +1 for Denver FAD :)
19:47:52 <tflink> any thoughts on when to do the FAD?
19:47:54 <pingou> puiterwijk: most app will switch when being re-written, so pushing
bodhi2 should make you happy ;-)
19:48:03 <nirik> lmacken: you said your place had some facilities?
19:48:05 <puiterwijk> pingou: :D
19:48:06 <lmacken> also, logging is another potential item. centralized app logs,
19:48:13 <nirik> oh yeah, logging++
19:48:15 <threebean> +1
19:48:15 * pingou thought end May beginning June
19:48:17 <lmacken> nirik: yeah, I can host a small FAD for sure
19:49:00 <nirik> pingou: might work.
19:49:26 <nirik> tflink: would that be something good from your side?
19:49:28 * masta looks in & lurks
19:49:50 <tflink> nirik: yeah, I'm definitely interested
19:50:03 <threebean> tflink: will you be at pycon again this year?
19:50:11 <nirik> when is pycon again?
19:50:15 * nirik goes to look
19:50:16 <threebean> aprilish?
19:50:23 <pingou> early April
19:50:27 <tflink> threebean: haven't decided yet, still need to figure that out
19:50:32 <nirik> ok, yeah, early apr
19:50:52 * willo is curious how people can assist remotely, if that's feasible
19:50:54 <threebean> cool, if you are and its no biggie, we could use that time to
discuss and lay out some things to think about before hand
19:51:04 <puiterwijk> I need to head out for now
19:51:20 <nirik> so, how about I toss up a fad page on the wiki for a
bodhi2/taskotron fad in late may/early june and we see from there if it will work out and
if we can get everyone we need there.
19:51:23 <threebean> willo: surely we can do some kind of teleconference. g+
hangout at least ;)
19:51:33 <adimania> +1 to willo
19:51:38 <adimania> and threebean
19:51:49 <nirik> yeah... we have #fedora-fad we use for irc to fads as they are
19:51:56 <nirik> and remote help is always great too.
19:52:18 <willo> all: great, long, long way to come for a day :)
19:52:45 <pingou> willo: that wouldn't be a single day ;-)
19:53:12 <willo> :)
19:53:14 <threebean> yeah, probably a whole weekend or maybe more? depends on
people's schedules once we know more what we want to accomplish
19:53:53 <nirik> yep. usually 3ish days
19:54:13 <abadger1999> plus travel days
19:54:33 <willo> is plan documented in wiki usually or outlined in mailing list
19:54:48 <nirik> usually setup in the wiki first and discussed on list.
19:54:50 <relrod> both usually
19:54:50 <willo> was wanting to check out what was done last time
19:54:54 <relrod> yeah
19:54:57 <willo> awesome
19:55:19 <nirik> look at our 2012 security fad...
19:55:26 <nirik> should be on wiki and on list. ;)
19:55:28 <nirik> #topic Open Floor
19:55:32 <nirik> any items for open floor?
19:55:45 <threebean> I just need to share that..
19:55:50 <threebean> ..for lunch, I ate only pickles.
19:56:06 * pingou needs to go pick up his dinner (pizza \ó/!)
19:56:28 <nirik> ha.
19:56:40 * willo wil be having hotdogs todays
19:56:48 <willo> several
19:57:06 <nirik> today's triva: does anyone know (without looking at wikipedia
or the like) what a blue eagle sign in a store would have meant? :)
19:57:37 <pingou> that the store owner is american? :)
19:57:50 <nirik> true.
19:58:03 <pingou> something to do w/ the army?
19:58:04 <nirik> and that their store was around in the us great depression. ;)
19:58:04 <willo> pingou: \o/
19:58:17 <pingou> didn't get that one
19:58:19 <nirik> anyhow, look it up if you like. ;)
19:58:20 <pingou> nirik: nice :)
19:58:25 <mariocav> O.o
19:58:30 <nirik> anything else? or will close out in a minute...
19:59:19 <nirik> thanks for coming everyone!
19:59:22 <nirik> #endmeeting