#fedora-meeting: Infrastructure (2014-08-14)
Meeting started by nirik at 17:59:55 UTC. The full logs are available at
* aloha (nirik, 17:59:55)
* LINK: https://fedoraproject.org/wiki/User:Adionditsak
* LINK: https://www.linkedin.com/pub/anders-aarvik/26/133/47b
* New folks introductions and Apprentice tasks (nirik, 18:02:51)
* Applications status / discussion (nirik, 18:10:09)
* copr update to seperate queues per arch, hopefully arm support
before too long. (nirik, 18:11:41)
* more optimizing on FMN ongoing (nirik, 18:11:54)
* some progress on new private openstack cloud. Hopefully more next
week. (nirik, 18:18:19)
* oddshocks and nirik to get fedimg uploading to old infrastructure
private cloud. (nirik, 18:21:00)
* LINK: http://jenkins.cloud.fedoraproject.org/job/fedora-rube/
* more rube tests needed (nirik, 18:22:28)
* Sysadmin status / discussion (nirik, 18:23:05)
* got fasClient fedmsg enabled and running from lockbox01. (nirik,
* new retrace machines being provisioned now. (nirik, 18:24:38)
* qa09/virthost-comm03 next to be provisioned. (nirik, 18:24:52)
* nagios/alerts recap (nirik, 18:26:06)
* LINK: http://tinyurl.com/ms5u5qm
* Many many thanks to Toshio (nirik, 18:33:00)
* Upcoming Tasks/Items (nirik, 18:35:14)
* LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/
* Open Floor (nirik, 18:38:59)
Meeting ended at 18:45:06 UTC.
Action Items, by person
People Present (lines said)
* nirik (88)
* threebean (16)
* adionditsak (15)
* lanica (9)
* mapyth (7)
* oddshocks (7)
* mirek-hm (7)
* lmacken (6)
* zodbot (5)
* danielbruno (5)
* bochecha (4)
* ootbro (3)
* adimania_ (3)
* abadger1999 (2)
* nmzaheer (2)
* lbazan (1)
* relrod (1)
* smooge (1)
* mdomsch (0)
* puiterwijk (0)
* pingou (0)
* dgilmore (0)
17:59:55 <nirik> #startmeeting Infrastructure (2014-08-14)
17:59:55 <zodbot> Meeting started Thu Aug 14 17:59:55 2014 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
17:59:55 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
17:59:55 <nirik> #meetingname infrastructure
17:59:55 <nirik> #topic aloha
17:59:55 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch
threebean pingou puiterwijk
17:59:55 <zodbot> The meeting name has been set to 'infrastructure'
17:59:55 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou
puiterwijk relrod smooge threebean
18:00:03 <smooge> aloha
18:00:07 <nirik> :)
18:00:08 <abadger1999> Greetings
18:00:09 <bochecha> hi everyone
18:00:20 * lmacken
18:00:28 <adionditsak> Greetings - i will copy paste a small introduction in 1
18:00:59 <adionditsak> I am a very new (1 week old) contributor/ambassador of
Fedora, and i look forward to get started for real. I just joined the FI apprentice group,
1 hour ago, and i am excited to dig into stuff. My name is Anders Aarvik, i am from
Denmark, and i am a young IT guy with 10 years of Linux usage, lately professionally as a
sysadmin (developer earlier). Tell me if you would like to know something else about me.
18:01:00 <adionditsak> ook forward to talk with you in the future.
18:01:00 <adionditsak> Sincerely,
18:01:01 <adionditsak> Anders Aarvik (adionditsak)
18:01:03 <adionditsak> https://fedoraproject.org/wiki/User:Adionditsak
18:01:05 <adionditsak> https://www.linkedin.com/pub/anders-aarvik/26/133/47b
18:01:20 * lanica is here for the infrastructure meeting.
18:01:46 * adimania_ is here
18:01:53 * mirek-hm is here
18:02:03 * danielbruno here
18:02:07 <ootbro> here
18:02:09 <oddshocks> here
18:02:10 * nmzaheer is here
18:02:12 * relrod here
18:02:28 <nirik> adionditsak: welcome. ;)
18:02:51 <nirik> #topic New folks introductions and Apprentice tasks
18:03:14 <ootbro> From the apprentice front .... the second diagram is now ready
for review at http://www.jt-sw.com/docs/index.nsf/docs/fpo-proxy
18:03:24 <nirik> adionditsak: were you more interested in sysadmin tasks or
18:03:31 <adionditsak> Thanks :-) Inserted a small intro further up
18:03:48 <adionditsak> Mostly system administration, but maybe a bit of development
in the future aswell.
18:03:57 <danielbruno> Hello guys, I contribute to Fedora for 5 ~ 6 years and I
intend to contirbute on infra group now
18:04:16 <nirik> ootbro: looks accurate to me. ;)
18:04:33 <nirik> adionditsak: cool. We have #fedora-apps for our application
18:04:35 <ootbro> thanks. this version has the extra notes that weren't in the
early preview I e-mailed you last week.
18:04:38 <nirik> danielbruno: welcome. :)
18:04:49 <danielbruno> nirik, thank you :)
18:05:08 <threebean> (here)
18:05:23 <adionditsak> Great :-) I will hang out in both channels then. What is the
fedora-ops for? Saw it under administration aswell.
18:05:54 <bochecha> adionditsak, operators of IRC channels
18:05:58 <nirik> adionditsak: thats unrelated to infrastructure. thats for irc
18:06:10 <adionditsak> Ah.
18:06:12 <danielbruno> more info about me:
18:06:16 <nirik> adionditsak: we also have #fedora-noc for sysadmin type things. It
also gets alerts and outage info, etc.
18:06:37 <adionditsak> Great - i will make a note on this
18:07:02 <nirik> danielbruno: you were interested in sysadmin tasks? or development?
or both? :)
18:07:18 <nirik> danielbruno: I can add you to our apprentice group to look around
after the meeting... see me in #fedora-admin.
18:07:39 <danielbruno> nirik, sysadmin tasks
18:08:07 <nirik> ok, great.
18:08:41 <nirik> welcome again to all the new folks. Do hang out in our channels and
ask questions anytime...
18:08:58 <adionditsak> Thank you :-)
18:09:07 <nirik> any other new folks?
18:09:51 <nmzaheer> Hi guys..i am just observing for now
18:10:06 <nirik> nmzaheer: ok. :) Feel free.
18:10:09 <nirik> #topic Applications status / discussion
18:10:26 <nirik> Any application development news/plans/discussion this week?
18:10:47 <mirek-hm> I upgraded Copr to new version. Main feature is that we know
have queues separate for each architecture, so Copr code is completly ready for new
arches. And adding those ARM is just matter of creating playbook for spinning up and
terminating (or reprovision) a machine.
18:10:53 <nirik> lmacken: what all do we need to do to get bodhi2 in staging?
shouldn't be too far should it?
18:10:58 <mirek-hm> s/know/now/
18:11:02 <threebean> I'm working on some further optimizations to fmn,
18:11:04 <nirik> mirek-hm: awesome!
18:11:12 <threebean> it's getting faster but still not yet where I'd like it
18:11:16 <lmacken> nirik: we'll need to port the ansible playbooks, since right
now bodhi.dev is setup by hand in a virtualenv with paster
18:11:34 <lmacken> bodhi.dev is also dead in the water right now with an expired
koji cert, and I haven't figured out how to fix it
18:11:41 <nirik> #info copr update to seperate queues per arch, hopefully arm
support before too long.
18:11:54 <nirik> #info more optimizing on FMN ongoing
18:12:21 <nirik> lmacken: ok. Happy to help with that. I think we can redo
bodhi01.stg with bodhi2 as soon as it's ready for it.
18:12:58 <lmacken> nirik: cool, sounds good. we're almost ready to dive into
that port, I just want to polish up the masher service and systemdify it
18:13:07 <nirik> fair enough.
18:13:18 <lmacken> or I guess we can just piggy-back on the moksha-hub service
18:13:19 <nirik> mirek-hm: did you get any further with cloud networking?
18:14:11 <mirek-hm> nirik: nope. I had to catch up my normal work after Flock
(rebuilding SCLs for el7)
18:14:17 <mirek-hm> hopefuly next week
18:14:26 <nirik> ok, no worries.
18:14:30 <mapyth> Hello Everybody!
18:14:30 <mapyth> I would like to update on the status of bugspad. As decided, I was
focussing on making the currently implemented
18:14:30 <mapyth> features more robust, and as bug free as possible. In the mean
time, my mentor has started doing a parallel work
18:14:31 <mapyth> in python. This would allow to get a performance metric, for
Python vs Golang. :)
18:14:43 <nirik> happily it seems like the old cloud has been reasonably stable for
18:14:51 <mirek-hm> yes
18:15:23 <threebean> mapyth: cool :)
18:15:23 <nirik> mapyth: cool. I still need to arrange a demo/talk with your mentor.
Hopefully tomorrow or early next week.
18:15:36 <threebean> I forget if we've asked already.. but is there a demo
instance of bugspad somewhere that we could play with?
18:15:52 <nirik> threebean: I think there is in the cloud... mapyth ?
18:16:05 <mapyth> nirik, threebean Please please check out, http://22.214.171.124/
18:16:28 <mirek-hm> nirik: last day on Flock I discovered
, which should easy writing
our playbook for our OS instance. asamalik tested it yesterday and it works fine. jruzicka
promised that he may try to package it for Fedora
18:16:41 <mapyth> nirik: Currently only basic features are done, and I am testing it
18:17:18 <nirik> mirek-hm: oh nice. I think I found that a while back and it
wasn't active, but looks like it has been recently now.
18:17:26 <nirik> mapyth: great
18:17:40 <mapyth> nirik: :)
18:18:19 <nirik> #info some progress on new private openstack cloud. Hopefully more
18:18:26 <nirik> any other application type news?
18:18:39 <threebean> mapyth: neat :)
18:18:53 * nirik was hoping abompard would be here and we could talk hyperkitty plans, but
will catch him when he's around.
18:19:07 <oddshocks> nirik: Is our internal openstack stuff ready to accept image
uploads from Fedimg?
18:19:29 <nirik> oddshocks: we could set that up on the old one... the new one is
still in a state of flux.
18:19:37 <oddshocks> cool.
18:19:38 <nirik> oddshocks: what all info do you need? a login/pass?
18:19:52 <oddshocks> nirik: yeah, whatever creds. i can check specifically right
18:20:16 <nirik> ok. I can get you those... ideally it would upload, but purge old
ones or else the list will get very large.
18:20:41 <oddshocks> nirik: yep just looks like auth user and auth pass for
18:20:52 <oddshocks> nirik: right. cool.
18:21:00 <nirik> #info oddshocks and nirik to get fedimg uploading to old
infrastructure private cloud.
18:21:04 <oddshocks> I don't know how openstack does image registration but
I'll figure it out
18:21:10 <threebean> oh, I almost forgot. we nearly have rube running in jenkins.
18:21:17 <nirik> excellent. ;)
18:21:18 <threebean> or, it's running.. but still failing on a test or two
18:21:20 <threebean> http://jenkins.cloud.fedoraproject.org/job/fedora-rube/
18:21:34 <nirik> what triggers runs? just timed?
18:21:46 <threebean> so, for people who don't know, rube is an automated test
suite for our staging webapps. it opens up firefox with selenium and tries to click
18:22:04 <threebean> getting people to contribute more tests would be really helpful
18:22:11 <threebean> nirik: yeah, just timed at the moment.
18:22:28 <nirik> #info more rube tests needed
18:23:05 <nirik> #topic Sysadmin status / discussion
18:23:12 <nirik> ok, on the sysadmin side...
18:23:13 <lmacken> threebean: can you hook me up with admin privs for it?
18:23:19 * threebean nods
18:23:25 <nirik> #info got fasClient fedmsg enabled and running from lockbox01.
18:23:43 <threebean> big win there :)
18:23:50 <nirik> Hopefully this will stop all those fasClient timeouts from when
things were running from cron.
18:24:03 <nirik> one nitpick... fedmsg shows me as running the playbook. :)
18:24:19 <nirik> or whatever user restarted the hub on lockbox01 last.
18:24:38 <nirik> #info new retrace machines being provisioned now.
18:24:52 <nirik> #info qa09/virthost-comm03 next to be provisioned.
18:25:34 <nirik> I'm not sure I want to start too much on the sysadmin side this
week, as we might go into freeze next tuesday.
18:25:54 <nirik> but the usual porting of stuff from puppet to ansible and rhel6 to
rhel7 is still ongoing.
18:26:06 <nirik> #topic nagios/alerts recap
18:26:16 * adimania_ needs to finish a couple of tickets before freeze
18:27:01 <nirik> I also closed out some older tickets...
18:28:05 <nirik> http://tinyurl.com/ms5u5qm
18:28:34 <nirik> hum, that might not be the right report.
18:29:41 <nirik> .tiny
18:29:46 <zodbot> nirik: http://tinyurl.com/ng2ouvc
18:30:05 <nirik> so, we have gotten a lot of packet loss on vpns, etc...
18:30:11 <nirik> hopefully thats cleared up
18:30:27 <nirik> busgateway01 is still producing a lot of alerts.
18:31:06 <nirik> threebean: can we bump the time on that busgateway monitoring some
18:31:36 * threebean nods
18:32:05 <threebean> not sure why its just nauncier that keeps bugging out there..
18:33:00 <nirik> #topic Many many thanks to Toshio
18:33:25 <nirik> As some of you know, abadger1999 is stepping back from Fedora
(hopefully only for a little while)
18:33:39 <nirik> Just thought I would mention it here and say many thanks to him for
all his work. ;)
18:33:42 <abadger1999> Thanks nirik :-)
18:34:34 <nirik> If theres anything fedora related you want from him, catch him
18:35:14 <nirik> #topic Upcoming Tasks/Items
18:35:15 <nirik> https://apps.fedoraproject.org/calendar/list/infrastructure/
18:35:27 <nirik> Anything upcoming anyone would like to note or schedule?
18:35:43 <nirik> we may be going into freeze next tuesday. We will see if we have a
test compose by then.
18:36:15 * lbazan here
18:36:45 <bochecha> nirik, does staging get frozen as well?
18:36:51 <nirik> also, there is another mass rebuild starting later today... so even
though we aren't in freeze, but carefull about any changes that could impact builders
18:37:05 <nirik> bochecha: nope. Staging never freezes. ;)
18:37:43 <bochecha> alright, just wondering if we'd be able to move forward with
puppet->ansible migrations during the freeze :)
18:37:47 <nirik> there's a script in ansible repo that lists whats frozen and
18:37:55 <nirik> yep. we should be able to no problem.
18:38:59 <nirik> #topic Open Floor
18:39:07 <nirik> ok, any items for open floor? questions, comments?
18:39:10 <adimania_> if I can recall, we might not be able to run playbooks on
18:39:26 <lanica> I've been working on trying to recreate the openid issue as
most recently seen on noc01 ... Ticket #4479
18:39:51 <lanica> I have yet to recreate it in my lab.
18:39:52 <nirik> adimania_: right. When a host is frozen, any changes to it need to
be sent to the list first and get at least 2 +1's from releng or sysadmin-main before
18:40:02 <nirik> lanica: yeah, it's a weird one. ;(
18:40:03 <adionditsak> Do we usually use Puppet or Ansible? I master Puppet, but i
am interested in Ansible. Have seen you talk about Ansible, but just logged into a server
with puppet agent.
18:40:16 <lanica> I'm not sure how long it takes to trigger or if perhaps it is
an issue that involves the front proxy servers
18:40:18 <nirik> adionditsak: both right now. We are moving from puppet to ansible.
18:40:28 <nirik> some hosts are still in puppet
18:40:34 <lanica> If anyone has more data or ideas...please add it to the ticket!
18:40:56 <adionditsak> Ok cool :-)
18:41:22 <lanica> I also do not have access to noc01 to try to collect more
information, which adds a degree of difficulty. If there are other servers that I can be
given access to that have had the issue in the past...that would be helpful.
18:41:51 <nirik> lanica: we don't actually have any other hosts using
mod_auth_openid. ;( This was the first one.
18:42:10 <lanica> Ok, I somehow missed that
18:42:24 <nirik> lanica: you should be able to login to log01 tho, and look under
/var/log/hosts/noc01*/ and see if there's any more info in logs. ;(
18:43:04 <nirik> doubt there will be, but possible I guess. ;)
18:43:10 <lanica> Ok, thanks. puiterwijk did collect the basic error messages that
seemed most relevant and sent them to me and I added it to the ticket. I'll dig
through there some more in case there are some hints.
18:43:29 <nirik> thanks for looking into it
18:43:35 <lanica> NP
18:44:15 <nirik> alright. Anything else? or shall we call it a meeting?
18:45:03 <nirik> Thanks for coming everyone. Do follow up over in #fedora-admin,
#fedora-apps and #fedora-noc. ;)
18:45:06 <nirik> #endmeeting