#fedora-meeting: Infrastructure (2013-01-24)
Meeting started by nirik at 19:00:04 UTC. The full logs are available at
* welcome y'all (nirik, 19:00:04)
* New folks introductions and Apprentice tasks. (nirik, 19:02:06)
* Applications status / discussion (nirik, 19:03:08)
* Sysadmin status / discussion (nirik, 19:31:35)
* Private Cloud status update / discussion (nirik, 19:49:37)
* Fudcon recap (nirik, 20:01:32)
* LINK: https://github.com/fedora-infra
* Open Floor (nirik, 20:10:49)
Meeting ended at 20:13:47 UTC.
Action Items, by person
People Present (lines said)
* nirik (96)
* skvidal (59)
* puiterwijk (40)
* abadger1999 (38)
* pingou (38)
* threebean (20)
* mdomsch (9)
* dgilmore (8)
* smooge (6)
* relrod (5)
* zodbot (4)
* suehle (4)
* mjg59 (3)
* lmacken (2)
* misc (2)
* maayke (1)
* Adran (1)
* ricky (0)
* CodeBlock (0)
19:00:04 <nirik> #startmeeting Infrastructure (2013-01-24)
19:00:04 <zodbot> Meeting started Thu Jan 24 19:00:04 2013 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
19:00:04 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:04 <nirik> #meetingname infrastructure
19:00:04 <zodbot> The meeting name has been set to 'infrastructure'
19:00:04 <nirik> #topic welcome y'all
19:00:04 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken
dgilmore mdomsch threebean
19:00:04 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch
nirik ricky skvidal smooge threebean
19:00:11 <nirik> who all is around for meeting?
19:00:15 <relrod> here
19:00:17 * skvidal is here
19:00:28 * puiterwijk is here (finally)
19:00:31 * lmacken
19:00:37 * maayke is here
19:00:58 * pingou
19:01:08 <dgilmore> hola
19:01:14 * abadger1999 here
19:01:57 <nirik> welcome everyone. ;)
19:02:04 <smooge> hgere
19:02:06 <nirik> #topic New folks introductions and Apprentice tasks.
19:02:15 <nirik> any new folks? or apprentice questions?
19:02:54 <nirik> going once...
19:03:08 <nirik> #topic Applications status / discussion
19:03:23 <nirik> tons of app stuff from fudcon. :) I tried to summarize some of it
on the list.
19:03:30 <pingou> and more to discuss
19:03:35 <nirik> if I left out something someone worked on, please chime in on
19:03:44 <nirik> yep. so, fire away...
19:04:07 <puiterwijk> hey,I have something
19:04:21 <puiterwijk> since FUDCon, I have been busy with a new OpenID provider for
19:04:23 <puiterwijk> FAS-OpenID
19:05:01 <puiterwijk> At this moment, it's stable and compatible with every
OpenID consumer I have found so far, so if anyone wants to test, please talk to me :)
19:06:09 <nirik> puiterwijk: I'll try and get a openid.stg instance setup
soon... we can ask more widespread testing then too.
19:06:15 <nirik> this _does_ have the groups handling?
19:06:24 <puiterwijk> yes, this DOES have groups extension
19:07:17 <nirik> ok.
19:07:45 <puiterwijk> this is also the basis for my 2factor authentication policy
19:08:09 <pingou> which basically brings the question of moving all our apps over to
19:08:23 <puiterwijk> in short: my idea (and Toshio's) was to make all of our
applications sign on using OpenID
19:08:30 * nirik nods.
19:09:05 <puiterwijk> applications are able to require the user to use 2nd factor to
access that app
19:09:14 <abadger1999> one ramification of this is that we'd be losing
shared-cookie SSO. But logging in to other apps would be trivial (wouldn't have to
retype password as the openid server would have a session cookie to authenticate you to
19:09:24 <dgilmore> one thing that would be nice is yubikey support in fedorahosted
19:09:49 <nirik> and getting away from mod_auth_pg
19:09:51 <nirik> would be very nice.
19:10:10 <puiterwijk> yes, the user wouldn't notice that he would be sent to
OpenID, as the provider returns him immediately if he sees the user is already logged on
19:10:49 <abadger1999> puiterwijk: We wouldn't have a "confirm you want to
send this set of information to this other application" anymore?
19:10:54 <pingou> the current implementation uses a time-out after 15min, the new
for the moment is valid for the time the browser is running
19:11:15 <puiterwijk> abadger1999: no, because we have a set of trusted roots (our
apps mostly) for which the user won't be asked to confirm. I already have that sytem
19:11:22 <abadger1999> we'd probably need to bump that up although unlimited is
19:11:32 <mjg59> /win 153
19:11:39 <dgilmore> mjg59: fail
19:11:40 <pingou> mjg59: oups :)
19:11:50 <relrod> and I thought 80 was a lot of windows. :(
19:11:56 <abadger1999> puiterwijk: excellent. The plan is fFor other apps (like
pypi), to still have that confirmation page, then?
19:11:59 <mjg59> Oh that's only halfway through.
19:12:05 <mjg59> (Sorry, laggy connection failure)
19:12:28 <puiterwijk> abadger1999: yes, so the skipping is only done for apps that
in the openid's trusted_root list
19:12:30 * nirik is ok with this plan in general. I want to ponder on it a bit more and
see if there's things I am not thinking of that would trip it up
19:12:31 <puiterwijk> (which we can configure)
19:12:31 <abadger1999> <nod>
19:12:41 <pingou> abadger1999: the 'white list' would be configure in the
configuration file of the application (iiuc)
19:12:45 <abadger1999> puiterwijk: what about CLI?
19:13:14 <puiterwijk> abadger1999: that would be harder, but I guess we'd have
to extend it
19:13:22 <pingou> puiterwijk: any idea how?
19:13:26 <abadger1999> Have to move to a model where we make a separate request to
login followed by requests for the information we want?
19:13:30 <dgilmore> puiterwijk: a lot of our apps have clis
19:13:41 <pingou> and we should keep them
19:13:46 <abadger1999> (using the session cookie we get back to auth those followup
19:13:49 <dgilmore> we have to keep them
19:13:56 <puiterwijk> abadger1999: yeah, something like that. maybe with a return
value saying "you need 2nd fa"
19:15:53 <abadger1999> k. I think we can make CLI work then... might just be
slightly less efficient under the hood.
19:16:05 <nirik> so, we need mod_auth_openid packaged again (for hosted, etc).
19:16:15 <puiterwijk> nirik: yeah, already busy on that one
19:16:21 <pingou> abadger1999: or via fedora-python?
19:16:28 <nirik> puiterwijk: my last attempt should still be around if you want it.
19:16:38 <puiterwijk> nirik: okay, will check that one
19:16:41 <nirik> also, can wiki use openid?
19:16:53 <nirik> I guess we can move things as we get them sorted out...
19:16:58 <puiterwijk> nirik: http://www.mediawiki.org/wiki/Extension:OpenID
19:17:24 <puiterwijk> yeah, I guess we can do that. that way, if we need more OpenID
extensions for some things, I can always add them
19:17:30 <skvidal> stupid question about openid
19:17:40 <skvidal> in the cases where we need to check for a single group
19:17:40 <puiterwijk> skvidal: yes?
19:17:55 <skvidal> is it possible to provide an openid url that ONLY works for that
subset of users in that group?
19:18:11 <pingou> like skvidal would like to do with copr atm :)
19:18:23 <skvidal> pingou: indeed
19:18:27 <puiterwijk> skvidal: I guess I could make that yeah
19:18:38 <pingou> and we will want that for cla+1 in most of our apps
19:18:39 <skvidal> then the user doesn't see the openid bits for fas-only auth
19:18:44 <skvidal> it just works
19:18:55 <pingou> unless we did as we do now and provide openid only for cla+1
19:19:01 <puiterwijk> skvidal: but on the other hand: you get group information, so
you might just say in your application to filter on that group
19:19:17 <skvidal> does that even make sense?
19:19:23 <puiterwijk> or I could develop an OpenID extension to say you require a
19:19:26 <skvidal> if there a nicer/better/more standards compliant way/ of doing
19:19:47 <skvidal> puiterwijk: I was thinking that a lot of apps might not be able
to do that w/o a lot of local hacking for groups
19:19:48 <nirik> and with badges and such we may be moving to providing openid/alias
19:20:05 <skvidal> puiterwijk: since most apps just think to get a yes or no from
19:20:09 <pingou> true there nirik
19:20:26 <puiterwijk> skvidal: as I said: I could just specify a new extension to
require a specific group
19:20:30 <skvidal> I was trying to avoid us having to write custom patches to apps
we don't maintain (like the wiki, for example)
19:20:32 <puiterwijk> skvidal: wouldn't be too hard
19:20:34 <nirik> if we do that we also need to be ready for people to use openid for
other things... like login to their local fedora machine. ;)
19:20:58 <pingou> \ó/
19:21:45 <puiterwijk> but maybe I could indeed define a new url for
19:22:08 <puiterwijk> or <group>.group.id.fp.o, we shoud discuss that
19:22:12 <nirik> anyhow, it looks like we all kinda like the idea... needs more work
and testing, but might end up saving us fas and application maint in the long run.
19:22:41 <abadger1999> <nod>
19:22:49 <skvidal> anyway - just a wacky thought
19:22:49 <skvidal> puiterwijk: just so we're clear - I think you're efforts
here look pretty damn good - I wasn't trying to nitpick with that question
19:23:04 <puiterwijk> skvidal: it's no problem :)
19:23:31 <puiterwijk> I understand :)
19:23:33 <pingou> puiterwijk: eta on 0.1.0 ?
19:23:49 <skvidal> well okay then
19:23:52 <puiterwijk> pingou: I guess today? maybe tomorrow?
19:23:57 * threebean is late, but likes this idea a lot
19:24:04 <puiterwijk> the only thing still needed is theming, and that's just a
bunch of HTML
19:24:05 <pingou> abadger1999: when is rolling the next fas release ?
19:24:21 * pingou proposes the FAN to puiterwijk
19:24:30 <abadger1999> pingou: I'm rolling a new python-fedora today. so fas
will probably get put off until tomorrow.
19:24:31 <pingou> Fedora Activity Night :p
19:24:39 <nirik> Smoother1rOgZ wasn't able to make it today but:
19:24:42 <nirik> Smoother1rOgZ> nirik: hey, I might be on my way home during
infra meeting so here's my input for apps update: we should have by now a FAS release
ready to update. toshio has howver some changes in stg which should bump fas to 0.8.17 as
19:24:44 <abadger1999> If only threebean would stop pushing new nad better code ;-)
19:24:54 <pingou> annoying threebean !
19:25:03 <nirik> heh.
19:25:29 <pingou> abadger1999: might be nice to roll another one after w/o the
19:25:38 <pingou> Smoother1rOgZ: ^
19:26:33 <nirik> puiterwijk: another dumb question: right now the openid provider
redirects to fas to authenticate right? Is there anything very fas specific in that?
19:27:02 <pingou> nirik: flask-fas under the hood
19:27:02 <puiterwijk> nirik: no, with about 10 lines changed, you can use any other
auth system that supports groups I guess
19:27:13 <threebean> :)
19:27:28 <abadger1999> pingou: <nod> -- once we have fas-openid deployed in
production we can remove the openid stuff from fas itself.
19:27:36 <nirik> ok, cool.
19:27:39 <pingou> abadger1999: nice
19:27:53 <nirik> Just thinking thats making us less 'fas specific' if we
decided to replace fas someday with something else.
19:28:12 <nirik> anyhow.
19:28:24 <nirik> any other apps news folks have? upcoming releases? cool things?
19:28:27 <abadger1999> flask seems pretty easy to code authentication methods for so
should be easier than currently.
19:28:45 <abadger1999> python-fedora release, fas release coming up soonest.
19:28:53 <abadger1999> packagedb release after a waiting period.
19:28:54 <pingou> pkgdb release in a bit :)
19:29:11 <threebean> /packages has been flipping out since saturday; the fix should
be ready to go out soon. hopefully tomorrow?
19:29:17 <abadger1999> nirik: I think we said two weeks from the time I announced
that features were going away.
19:29:21 <nirik> yep
19:29:25 <abadger1999> Cool.
19:29:36 <threebean> dgilmore and I are talking about hooking koji up to fedmsg,
again hopefully tomorrow
19:29:45 <abadger1999> threebean: yeah -- Does that only need the pyhton-fedora
update or something more aswell?
19:30:15 <threebean> abadger1999: just that update, but I also have to write an init
file and do the release dance
19:30:28 <abadger1999> k
19:31:01 <nirik> awesome. :)
19:31:26 * nirik will move on then if nothing else pressing...
19:31:35 <nirik> #topic Sysadmin status / discussion
19:31:47 <nirik> I'm going to schedule some outages next week... tuesday and
19:32:00 <nirik> going to do updates/mass reboots of class a / b machines.
19:32:15 <nirik> and also move our iscsi storage. (That will be so fun!)
19:32:30 <mdomsch> made good progress on MM at FUDCon. Looking forward to taking
more patches from abadger1999, pingou, and smooge
19:32:30 <mdomsch> :-)
19:32:33 <mdomsch> want to get that into staging soon and find out what else I
19:32:50 <pingou> mdomsch: awesome!
19:33:00 <smooge> I am looking at taking the syslog patch we did and add it to
19:33:25 <abadger1999> Yay!
19:33:39 <smooge> so I can track when servers are out of order so we can see it
19:34:07 <mdomsch> snapmirror to download-i2 running again
19:34:31 <mdomsch> smooge yea!
19:34:47 <nirik> yeah, I moved download-i2 to point back to download-ib01 for now.
will change it back when the rdu ones are up to date.
19:35:15 <smooge> and galgoci said we needed to change the IPs anyway on the RDU
boxes because he gave out the wrong ones
19:35:17 <mdomsch> nirik: care to comment on the new hotfix policy?
19:35:22 <mdomsch> I think it's a great idea
19:36:07 <skvidal> smooge: nice
19:36:15 <nirik> oh yeah...
19:36:18 <mdomsch> check in full files, then check in the patch
19:36:26 <mdomsch> so we can easily see the patches
19:36:28 <nirik> made a sop:
19:39:03 * nirik looks at other sysadmin stuff...
19:39:12 <puiterwijk> nirik: maybe discuss my suggestion?
19:39:33 <puiterwijk> the suggestion to make noc able to use run-puppet nowait on
all servers they have access to?
19:39:36 * nirik is somewhat distracted by ongoing dns outage issue.
19:39:55 <nirik> yeah, I'm not opposed to that
19:39:56 <puiterwijk> as right now, I have to wait half an hour for puppet to run on
19:40:23 <puiterwijk> (like last weekend when I was watching the servers, I had to
wait a lot for memcached ...)
19:40:47 <abadger1999> +1
19:40:56 * threebean likes it too
19:40:59 <nirik> when we go to ansible we were looking at making it run only on
19:41:13 <skvidal> nirik: hmmm - maybe a sudo entry on lockbox01
19:41:21 <dgilmore> im ok with that as long as sysadmin-noc has no access to any of
the build/releng boxes
19:41:43 <skvidal> nm
19:41:43 <skvidal> sorry
19:41:53 <nirik> dgilmore: I don't think they do.
19:42:00 <nirik> can doublecheck.
19:42:02 <puiterwijk> nope, we don't
19:42:06 <puiterwijk> (we as in noc)
19:42:29 <puiterwijk> noc doesn't have access to those. but I suggested only
sudo run-puppet nowait access to noc)
19:42:38 <nirik> ok, I'll make that sudoers change after the meeting. ;)
19:42:45 <puiterwijk> ok, thanks :)
19:42:47 <nirik> anything else sysadmin wise?
19:43:19 * threebean has one
19:43:21 <threebean> kinda
19:43:37 <threebean> misc re-raised the idea of sending nagios notifications out
19:43:39 <nirik> sure, fire away
19:43:53 <threebean> we discussed it over the summer and decided that it was unwise
to attach nagios in any way to it.
19:44:34 <threebean> now that we've seen fedmsg working (most of the time), is
anyone interested in talking about it again or trying it out?
19:44:53 <nirik> well, not sure...
19:45:05 <threebean> it would only make sense in a supplementary capacity. notifs
via email, sms, and fedmsg.
19:45:26 <nirik> I'd say we should look at doing that as part of a larger effort
to rework nagios, which I intend to work on
19:45:38 <abadger1999> Hard deps would definitely still be out. But we do have code
to add fedmsg such that it doesn't fail if fedmsg is not available
19:45:56 * abadger1999 remembers to add that to the AppBestPractics page
19:46:00 <nirik> right, it would be informational, not us acting on it.
19:46:18 <nirik> and if we decided to act on it, we would query nagios directly to
make sure that was the case.
19:46:25 * threebean nods
19:46:39 <skvidal> a couple of minor things
19:47:16 <nirik> trust, but verify. or... don't trust, and verify. ;)
19:47:42 <skvidal> threebean: we've still not seen fedmsg under load, have we?
19:47:56 <skvidal> ah
19:48:18 <threebean> skvidal: not real load. I've done some dummy tests.
19:48:48 * Adran wanders in. [sorry about being late.]
19:49:16 <skvidal> okay
19:49:30 <nirik> ok, any other sysadmin stuff? shall we move on?
19:49:37 <nirik> #topic Private Cloud status update / discussion
19:49:39 <skvidal> I'm a LITTLE concerned about our nagios info being public
19:49:44 <nirik> any cloudy stuf?
19:49:53 <skvidal> since putting it on fedmsg would make it available to everyone
19:49:53 <skvidal> but not massively so
19:50:02 <nirik> skvidal: true. it's sysadmin now right?
19:50:09 <threebean> yeah.. I have to auth for
19:50:17 <skvidal> nirik: sysadmin-noc, isn't it?
19:50:28 <pingou> skvidal: sysadmin
19:50:32 <pingou> I can log in
19:50:32 <skvidal> okay
19:50:53 <misc> maybe having it public would permit to external people to say
"we know, this is broken", instead of having people asking over on irc to the
few one able to fix the issue :)
19:50:54 <skvidal> so it's not a high threshhold - ubt it 's a far cry from
19:50:55 <nirik> yeah, and we send them to sysadmin. ;)
19:50:58 <skvidal> misc: no
19:51:03 <skvidal> misc: here's what will happen
19:51:07 <skvidal> they'll see it is a problem
19:51:09 <skvidal> and come tell us
19:51:19 <misc> ah, yeah, maybe too :)
19:51:25 <skvidal> it'll be like the most laggy paging system in the world
19:51:45 <threebean> ha :P
19:51:48 <threebean> distributed paging
19:51:53 <threebean> someone is *sure* to tell us :)
19:52:26 <nirik> :)
19:52:28 <skvidal> anyway
19:52:33 <relrod> skvidal: We let anyone idle in sysadmin-noc, doesn't that
effectively make it available to anyone, anyway?
19:52:38 <skvidal> relrod: no
19:52:43 <skvidal> 1. we don;'t let ANYONE in
19:52:47 <relrod> #sysadmin-noc
19:52:48 <relrod> sorry
19:52:55 <skvidal> oh
19:52:59 <skvidal> and still no
19:53:04 <skvidal> there is social pressure in that channel
19:53:08 <skvidal> it exists in all places
19:53:13 <skvidal> there is no pressure to listen to fedmsg logs
19:53:15 <skvidal> anyway
19:53:19 <skvidal> do not let me concerns be a blocker
19:53:34 <skvidal> I just wanted to put them out there b/c when/if things go
horribly wrong I enjoy saying I told you so ;)
19:53:39 <nirik> ha. ;)
19:53:54 <nirik> I think we can think about it in a new nagios world... it may or
may not make sense then.
19:53:59 <skvidal> agreed
19:54:12 <threebean> cool
19:55:12 <nirik> anyhow, on to clouds. ;)
19:55:20 <skvidal> so the minor items I had were about ansible playbooks
19:55:34 <nirik> I setup tflink with an openstack login/account and he's playing
with it to see how it will work for qa needs.
19:55:36 <skvidal> I got to test the host-reboot playbook in anger this week and it
worked as intended
19:55:42 <skvidal> I see
19:55:45 * skvidal moves along
19:55:59 <nirik> sorry, we can go back too. ;)
19:56:08 <smooge> my next nagios tag will output to fedmsg the password hash of
people who have problems logging in. That way they can grab their hash and see if that
helps them remember it.
19:56:31 <smooge> sorry slow type
19:57:05 <pingou> ??
19:57:14 <skvidal> pingou: pretty sure smooge was kidding
19:57:22 <pingou> I surely hope so! :D
19:57:42 <nirik> so, we are adding more cloud stuff... we should keep working on
finishing up stuff before we call it production.
19:57:43 <pingou> skvidal: I was like "wat" :)
19:57:56 <nirik> I think we can hash out much of whats left before too long.
19:58:40 <skvidal> everytime we add something persistent in the cloud I add it to
ansible - provided that the provisioning process there remains the same - and we can
migrate the data from the storage controller on the euca instance - we can move it about
19:59:20 <nirik> cool. I guess we need to hook up openstack to ansible as well...
20:00:03 <nirik> we can set them up with fasClient and two factor sudo anytime.
20:00:16 <nirik> anyhow, we will keep working on things there.
20:00:17 <skvidal> nirik: the only thing we need to connect openstack to ansible
20:00:18 <skvidal> afaict
20:00:23 <skvidal> is a functional, ssl'd ec2 api
20:00:26 <skvidal> that's it
20:00:38 <nirik> ok. I thought we had that, or was that not ssled?
20:00:45 <skvidal> not ssl'd
20:00:47 <skvidal> right
20:01:08 <nirik> ok
20:01:29 <nirik> ok, can poke at it more.
20:01:32 <nirik> #topic Fudcon recap
20:01:38 <nirik> it was great to meet up with everyone!
20:01:54 <nirik> anything people want to followup after fudcon ?
20:02:00 <abadger1999> Yeah -- and this year, it felt like we had a big healthy team
20:02:04 <nirik> we didn't decide a FAD, but we have no shortage of choices.
20:02:31 <abadger1999> We started discussing app best practices and lmacken created
this page => https://fedoraproject.org/wiki/Infrastructure/AppBestPractices
20:02:43 <abadger1999> We've been adding things we thought about to it since
20:02:55 <abadger1999> Feel free to peruse, add, and discuss.
20:03:07 <nirik> cool.
20:03:07 <threebean> :)
20:03:11 * pingou 'd love a app FAD
20:03:15 <lmacken> oo, we should probably mention kitchen in there :)
20:03:30 <abadger1999> Good idea.
20:04:34 <abadger1999> We never got around to graphining out all of our dependencies
at fudcon but nirik, dgilmore, and I started thinking about it in terms of how it related
20:04:36 <suehle> If anybody's here for the marketing team meeting, let's
move over to #fedora-meeting-1 so these guys can finish.
20:04:54 <nirik> suehle: sorry, we could give you folks the room if you like.
20:04:58 <nirik> we are running slow today. ;(
20:05:13 <suehle> nirik, no problem at all!
20:05:24 <nirik> lets close out and move discussion over to #fedora-admin. ;)
20:05:27 <suehle> keep being productive and/or slow :)
20:05:29 <abadger1999> I'll send something to admin@ hopefully tomorrow
20:05:53 <skvidal> suehle: thppppt
20:06:44 <nirik> abadger1999: cool. I know we need to discuss more, we didn't
come to too many conclusions.
20:06:53 <abadger1999> We moved some project hosting from fedorahosted to github.
Pull requests and the code review you can do with it has already been a big help.
20:06:53 <dgilmore> abadger1999: indeed its a complex entanglement
20:07:11 <suehle> skvidal, nirik was the one who said you were being slow, raspberry
20:07:32 * dgilmore refuses to use github
20:07:37 <abadger1999> https://github.com/fedora-infra
20:09:01 <nirik> would it be possible/easy to also hook hosted into them?
20:09:17 <abadger1999> pingou or threebean ^ ?
20:09:31 <pingou> we should be able to have a post-hook script
20:09:32 <abadger1999> nirik: just as a mirror of the github repo you mean?
20:09:38 <pingou> which mirrors to hosted
20:10:00 <pingou> someone also mention that we can ask github to be the frontend of
projects hosted elsewhere
20:10:05 <pingou> it's not in the UI but can be asked
20:10:11 <pingou> that might be worth looking into
20:10:17 <nirik> yeah, that might be nice.
20:10:33 <nirik> then hosted gets all the changes and people can contribute there or
github for those projects that want
20:10:49 <nirik> #topic Open Floor
20:10:51 <pingou> yup
20:10:59 <nirik> anything for open floor? we are over time... ;(
20:11:06 <abadger1999> <nod> although we should think about if we want to do
code review of all changes.
20:11:18 <abadger1999> If so, going through github would be better than going
20:11:53 <pingou> abadger1999: the UI would be gh and the git on hosted
20:12:53 <nirik> ok, anything else/
20:12:54 <nirik> ?
20:13:45 <nirik> ok, thanks for coming everyone!
20:13:47 <nirik> #endmeeting