#fedora-meeting: Infrastructure (2014-08-28)
Meeting started by nirik at 18:00:11 UTC. The full logs are available at
* aloha (nirik, 18:00:11)
* New folks introductions and Apprentice tasks. (nirik, 18:01:38)
* Freeze reminder (nirik, 18:12:14)
* Applications status / discussion (nirik, 18:13:52)
* fedocal, pkgdb2 bugfix releases this week (nirik, 18:17:16)
* nuancier release with lots of changes. (nirik, 18:17:27)
* fedimg work ongoing (nirik, 18:17:39)
* Sysadmin status / discussion (nirik, 18:23:42)
* pkgs01.stg now in ansible and running rhel7 (nirik, 18:24:53)
* fixed a bunch of hosts monday that are still in puppet that were
stuck. ;( (nirik, 18:25:52)
* nagios/alerts recap (nirik, 18:27:39)
* Upcoming Tasks/Items (nirik, 18:33:17)
* LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
* Open Floor (nirik, 18:36:03)
Meeting ended at 18:39:52 UTC.
Action Items, by person
People Present (lines said)
* nirik (70)
* pingou (32)
* oddshocks (18)
* danielbruno (7)
* mirek-hm (7)
* lanica (6)
* Neldogz (5)
* zodbot (5)
* smooge (3)
* vgologuz (3)
* threebean (2)
* lmacken (1)
* bwood09 (1)
* danofsatx (1)
* relrod (1)
* puiterwijk (0)
* abadger1999 (0)
* mdomsch (0)
* dgilmore (0)
18:00:11 <nirik> #startmeeting Infrastructure (2014-08-28)
18:00:11 <zodbot> Meeting started Thu Aug 28 18:00:11 2014 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
18:00:11 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
18:00:11 <nirik> #meetingname infrastructure
18:00:11 <zodbot> The meeting name has been set to 'infrastructure'
18:00:11 <nirik> #topic aloha
18:00:11 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch
threebean pingou puiterwijk
18:00:11 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou
puiterwijk relrod smooge threebean
18:00:17 * pingou here
18:00:21 * bwood09 is here
18:00:32 * oddshocks here
18:00:34 * danielbruno here
18:00:35 * lmacken
18:00:51 * mirek-hm is here
18:00:58 <Neldogz> neldogz is here
18:01:30 * threebean is here
18:01:36 <nirik> cool. Lets go ahead and dive on in then...
18:01:38 <nirik> #topic New folks introductions and Apprentice tasks.
18:01:45 <nirik> any new folks want to intrduce themselves?
18:01:53 <nirik> or apprentices with questions/comments/ideas?
18:02:05 <smooge> here
18:02:21 <nirik> hey smooge
18:02:31 <danielbruno> I have some questions about the Apprentice tasks
18:02:35 * danofsatx present for accounting purposes
18:02:38 <nirik> danielbruno: fire away. ;)
18:02:41 <Neldogz> hello by any chance are there any openings in the Fedora
Infrastructure team for volunteers? I have a few hours a week that I would love to
dedicate to this team and the Fedora Project.
18:03:02 * lanica is here for the infra meeting.
18:03:29 <nirik> Neldogz: see
and yes, always looking for
folks to help out... are you more interested in sysadmin tasks? or application
18:03:30 <danielbruno> so, I read the documentations.. and I would like to know a
good point to start
18:03:45 <danielbruno> if I have to ask in the channel
18:04:30 <Neldogz> hi nirik; mostly in systems administration. I do have experience
with RHEL and Fedora. I would love to learn more about how these work in a corporate
environment. I am pretty good with monitoring, documentation and up for anythign that is
18:04:34 <nirik> danielbruno: well, best bets are to look at the easyfix tickets and
pick something you are interested in and start working on it... and/or hang out in our
channels and when you see an interesting to you discussion chime in and ask if you can
help out. ;)
18:05:14 <nirik> if there's specific areas you are interested in we might be
able to come up with some more targeted things... :)
18:05:38 <danielbruno> great! :)
18:05:45 <nirik> Neldogz: ok, welcome. we have an apprentice program (see link off
that getting started page) if you are interested in it, I can add you after the meeting...
see me in #fedora-admin.
18:06:06 <Neldogz> great! would love to join fi-apprentice
18:06:12 <Neldogz> see you in #fedora-admin
18:06:27 <nirik> cool. ;)
18:06:58 <nirik> I will also see about filing some more easyfix tickets...
haven't done any new ones for a bit.
18:07:28 <danielbruno> nirik, I have looked at the easyfix and i'm interested in
18:07:28 <vgologuz> hi all
18:08:08 <nirik> danielbruno: cool. Do comment in those tickets and/or ask the
person who filed them if they are around on irc...
18:08:15 <nirik> morning vgologuz
18:08:34 <vgologuz> I've recently joined Copr project, and would like (in some
time) to learn about fedora infrastructure
18:08:41 <danielbruno> nirik, ok.. I will :)
18:09:01 <nirik> vgologuz: great. :) I can also add you to our apprentice group if
you like... see me after the meeting in #fedora-admin.
18:09:37 <vgologuz> nirik, great )
18:10:05 <mirek-hm> I will try to learn vgologuz as much as I know about
ansible.git, playbooks and FedoraCloud
18:10:25 <mirek-hm> s/learn/teach/
18:12:04 <nirik> great.
18:12:06 * nirik just had a laptop lockup. sorry for the delay.
18:12:14 <nirik> #topic Freeze reminder
18:12:44 <nirik> So, just a reminder that we are freeze now. Any frozen hosts should
not be modified without a patch with 2 +1s from sysadmin-main or releng.
18:12:58 * pingou was the first one to forgot about it
18:13:04 <pingou> it has been so long since the last one :)
18:13:37 <nirik> yeah, I know. ;)
18:13:44 <nirik> see the mail on the list for more details. ;)
18:13:52 <nirik> #topic Applications status / discussion
18:14:04 <nirik> any applications news/status this week?
18:14:08 <lanica> Everything is frozen besides stg?
18:14:10 <lanica> Sorry for the lateness
18:14:34 <pingou> new fedocal released earlier this week, bug fix mainly
18:14:50 <pingou> new pkgdb2 released as well, bugfix again
18:14:59 <pingou> new nuancier released, quite some changes on that one :)
18:15:15 <oddshocks> Fedimg is installed on fedimg01.stg. Working out some Comcast
internet problems this moment (I'm currently tethered to my phone 'net), but once
they're resolved I'm gonna find out if it's properly doing the thing
18:15:20 <pingou> and elections just got a new election type: irc which offers
18:15:29 <pingou> work from nerdsville
18:15:35 <oddshocks> (finally got permissions stuff with my sudo access to
fedimg01.stg worked out.. I think)
18:15:47 <oddshocks> /s/i think//
18:16:01 <pingou> oddshocks: ansible?
18:16:50 <nirik> lanica: there's a script in ansible called 'freezelist'
it will show the ones frozen/not.
18:17:02 <nirik> there are some non stg hosts that also aren't frozen.
18:17:03 <lanica> nirik: thanks!
18:17:16 <nirik> #info fedocal, pkgdb2 bugfix releases this week
18:17:27 <nirik> #info nuancier release with lots of changes.
18:17:39 <nirik> #info fedimg work ongoing
18:17:41 <oddshocks> pingou: what about it? I'm not 100% clear on what the issue
was, nirik and smooge resolved it for me
18:18:07 <oddshocks> pingou: i didn't install fedimg with ansible yet, if
that's what you're asking. just `rpm -i`'d the rpm
18:18:19 <pingou> oddshocks: that was my question :)
18:18:21 * relrod checks in late, sorry
18:18:27 <oddshocks> I need to figure this ansible stuff out though. I have little
to no experience with it sad to say
18:18:31 <oddshocks> relrod: hey
18:18:39 <pingou> oddshocks: installing it via a playbook might have solved the
problem there :)
18:18:40 <nirik> oddshocks: we can walk you through it.
18:18:45 <oddshocks> nirik: awesome, thanks :)
18:18:47 <oddshocks> pingou: ahh
18:19:00 <smooge> oddshocks, none of us know how it works. I have 3 voudou dolls
which help me
18:19:05 <oddshocks> nirik: pingou: TBH I'm not even 100% clear on which things
I should be doing with ansible, and what it can do for me
18:19:06 <pingou> oddshocks: feel free to ping wrt ansible
18:19:09 <oddshocks> smooge: haha
18:19:14 <oddshocks> pingou: cool
18:19:19 <pingou> oddshocks: install rpm, configuration file
18:19:41 <pingou> oddshocks: basically everything that's needed to drop the VM
and build a new one working out-of-the-box
18:20:05 <pingou> oddshocks: getting one of those woudou dolls from smooge is a good
18:20:28 <nirik> basically if we had to reinstall the machine today, ansible would
let us get it back to the same exact working state.
18:20:33 <oddshocks> pingou: ah that makes sense.
18:20:44 <oddshocks> nirik: right, so yeah just installing the fedimg rpm and
putting fedimg.cfg in /etc/
18:20:54 <pingou> nirik: are we doing cloud on kvm? :)
18:20:55 <oddshocks> and likely starting the fedimg relay if it doesn't do so
18:21:10 <oddshocks> nirik: I'll also ping you soon this week about internal
openstack uploading if you'd like
18:21:13 <pingou> oddshocks: sounds nice and simple, shouldn't be too hard to do
with ansible :)
18:21:17 <nirik> oddshocks: sounds good.
18:21:24 <nirik> pingou: our cloud uses kvm, yes?
18:21:55 <mirek-hm> yes, OS use kvm
18:21:56 <pingou> nirik: I meant more our stg/prod boxes, we're always
"ready" to turn one down and rebuild one, sounds cloudy :)
18:22:16 <nirik> pingou: kinda, but more persistent I think...
18:22:29 <pingou> hopefully :)
18:22:35 <nirik> and more decentralized
18:22:48 <pingou> that's very true
18:23:06 <nirik> any other application news?
18:23:24 <nirik> I'm gonna try and ping abompard next week when he's back
about moving more forward on hyperkitty
18:23:35 * pingou looks forward
18:23:42 <nirik> #topic Sysadmin status / discussion
18:23:50 <nirik> lets see...
18:24:19 <nirik> I got bochecha's pkgs migration to ansible commited and made a
18:24:31 <nirik> There's more work we need to do on it, but I think it's
coming along nicely.
18:24:53 <nirik> #info pkgs01.stg now in ansible and running rhel7
18:25:08 <pingou> w/ it we're moving to gitolite3
18:25:39 * nirik nods.
18:25:40 <pingou> is that the remaining work you were thinking about or are there
also other things?
18:25:52 <nirik> #info fixed a bunch of hosts monday that are still in puppet that
were stuck. ;(
18:26:01 <nirik> yeah, mostly gitolite3 and lots of testing.
18:26:08 <pingou> +1 on testing
18:26:29 <nirik> cgit isn't working right for some reason yet.
18:26:30 <pingou> note: the gitolite configuration cron/task isn't on (or
18:27:02 <nirik> yeah.
18:27:25 <nirik> So, given the freeze we can work on stg stuff and documentation and
18:27:30 <pingou> \ó/
18:27:39 <nirik> #topic nagios/alerts recap
18:27:40 <mirek-hm> I proceed with Openstack installatation. networking is hopefully
fixed (although after reset it need manual intervention via console :( ) and I'm
currently fighting with one problem in packstack.
18:27:59 <nirik> mirek-hm: great. ;) so it's getting close?
18:28:05 <mirek-hm> yes
18:28:19 <pingou> I've been working with releng to drop the pkgdb2branch script
18:28:20 <mirek-hm> it is my priority now
18:28:59 <nirik> awesome. we still need to hook in storage... but hopefully that
won't be too bad
18:29:12 <pingou> the idea is to replace that script by another that checks all git
for their branches vs the branches they should have according to pkgdb and adjust the git
18:29:30 <pingou> so that's one more thing changing with the move of pkgs01 to
18:30:06 <nirik> .tiny
18:30:14 <zodbot> nirik: http://tinyurl.com/m3v3a5x
18:30:59 <nirik> most of our alerts continue to be network related... but I've
found that it's likely openvpn getting saturated.
18:31:13 <nirik> I've not isolated what traffic is causing it yet, but will keep
18:31:21 <pingou> threebean: and I are thinking about giving a trial for
datanommer/datagrepper on a mongodb backend, that might help with datagrepper's
18:32:17 <nirik> ok. I wonder also if moving that db to rhel7 and newer postgres
18:32:29 <pingou> could be also
18:33:17 <nirik> #topic Upcoming Tasks/Items
18:33:17 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:33:26 <nirik> anything upcoming anyone would like to schedule or note?
18:34:00 <smooge> not from me
18:34:10 <threebean> pingou: yeah, we should really follow up with yograterol on
that. he has actual experience with mongo and is interested in helping with FI
18:34:57 <nirik> we should make sure we know what we are getting into before we move
to it, but trying things seems like a great first step .
18:35:30 <pingou> threebean: +1 but I haven't seen him on irc recently :(
18:36:03 <nirik> #topic Open Floor
18:36:10 <nirik> anyone have anything for open floor?
18:36:20 <nirik> questions, ideas, favorate cartoons?
18:37:38 <oddshocks> I like The Boondocks
18:37:42 <lanica> noc01/nagios mod_auth_openid issue -- looks to be a cobination of
a bug in mod_auth_openid and libopkele. One patch, already in existance for
mod_auth_openid fixes part of the issue. Once I can find some time, I can do some more
testing and validate the fix for libopkele. I probably need to submit a bug report for
libopkele if it does not already exist.
18:37:50 <lanica> DuckTales perhaps?
18:38:10 <nirik> lanica: awesome. Thanks for tracking that down... it's an
anoying bug. ;)
18:38:17 * nirik likes bob's burgers.
18:39:07 <nirik> alright, will close out in a minute if nothing more.
18:39:49 <nirik> Thanks for coming everyone!
18:39:52 <nirik> #endmeeting