============================================
#fedora-meeting: Infrastructure (2014-03-06)
============================================
Meeting started by nirik at 19:00:03 UTC. The full logs are available at
http://meetbot.fedoraproject.org/fedora-meeting/2014-03-06/infrastructure...
.
Meeting summary
---------------
* welcome everyone (nirik, 19:00:03)
* New folks introductions and Apprentice tasks (nirik, 19:02:06)
* Meeting time (DST) (nirik, 19:06:05)
* meeting to move ahead an hour next week (nirik, 19:08:36)
* Applications status / discussion (nirik, 19:08:52)
* LINK:
https://admin.stg.fedoraproject.org/accounts stg is updated
(pingou, 19:09:23)
* new fas release out soon, stg updated (nirik, 19:12:18)
* pkgdb2-cli is close to ready and pkgdb2 hopefully to stg next week
(nirik, 19:12:39)
* LINK:
https://github.com/fedora-infra/fas/blob/develop/updates/fas2-0.8.9.1_to_...
(pingou, 19:13:04)
* LINK:
https://apps.stg.fedoraproject.org/calendar/fedora-meeting/
check the month calendar on the side (pingou, 19:15:21)
* bodhi01 / 02 production instances added to the bodhi app pool
(nirik, 19:16:46)
* Sysadmin status / discussion (nirik, 19:21:56)
* LINK:
https://fedoraproject.org/wiki/Infrastructure/PuppetToAnsibleMigration
(nirik, 19:22:42)
* paste and smtp-mm migrated over recently (nirik, 19:24:42)
* LINK:
https://fedoraproject.org/freemedia/FreeMedia-form.html is
freemedia (nirik, 19:26:49)
* updates applied all around yesterday/day before. (nirik, 19:28:14)
* ansible 1.5 came out recently. Some nice improvements. (nirik,
19:29:24)
* Upcoming Tasks/Items (nirik, 19:41:05)
* LINK:
https://apps.fedoraproject.org/calendar/list/infrastructure/
(nirik, 19:41:05)
* LINK:
https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014
(nirik, 19:42:16)
* Open Floor (nirik, 19:44:17)
* LINK:
http://paste.fedoraproject.org/83130/94135484/ (nirik,
19:51:32)
Meeting ended at 20:09:35 UTC.
Action Items
------------
Action Items, by person
-----------------------
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* nirik (95)
* pingou (34)
* smooge (20)
* willo (9)
* abadger1999 (6)
* purplex (6)
* danofsatx-work (5)
* axil42 (5)
* zodbot (4)
* johe (4)
* mirek-hm (4)
* tflink (3)
* lmacken (2)
* webpigeon (2)
* relrod (2)
* lbazan (1)
* janeznemanic (1)
* puiterwijk (0)
* threebean (0)
* mdomsch (0)
* dgilmore (0)
--
19:00:03 <nirik> #startmeeting Infrastructure (2014-03-06)
19:00:03 <zodbot> Meeting started Thu Mar 6 19:00:03 2014 UTC. The chair is nirik.
Information about MeetBot at
http://wiki.debian.org/MeetBot.
19:00:03 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:03 <nirik> #meetingname infrastructure
19:00:03 <nirik> #topic welcome everyone
19:00:03 <nirik> #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch
threebean pingou puiterwijk
19:00:03 <zodbot> The meeting name has been set to 'infrastructure'
19:00:03 <zodbot> Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou
puiterwijk relrod smooge threebean
19:00:15 * lmacken
19:00:22 <nirik> hey everyone.
19:00:31 * johe is here
19:01:01 * pingou
19:01:04 * tflink is lurking
19:01:09 <axil42> hello :)
19:01:49 <webpigeon> Hello :)
19:01:57 <nirik> ok, lets go ahead and dive in...
19:01:58 <janeznemanic> hi
19:02:06 <nirik> #topic New folks introductions and Apprentice tasks
19:02:09 <purplex> Hello everyone!
19:02:15 <nirik> any new folks want to introduce themselves?
19:02:22 <nirik> Or apprentices with questions or comments?
19:02:33 <axil42> yeap :)
19:02:42 <purplex> Introducing myself
19:02:47 * relrod here, sorry
19:03:06 * abadger1999 extricates from other meeting
19:03:07 <nirik> welcome axil42 and purplex. Do give a short sentence introduction
if you can...
19:03:11 <smooge> hello
19:03:12 <purplex> I'm Bhumish, currently a student. I'm into system admin
and a Fedora-lover.
19:03:50 <pingou> abadger1999: good luck :)
19:04:15 <axil42> I'm Achilleas also student, currently Fedora packager and want
to get into sysadmin tasks :)
19:04:24 <nirik> purplex / axil42: are you more interested in sysadmin type tasks or
application development/coding?
19:04:32 * mirek-hm joins
19:04:51 <purplex> I'm interested in sysadmin tasks. Currently I'm exploring
FUNC.
19:04:53 * willo here, apologies
19:04:59 <axil42> me more on sysadmin, like I said in my introduction mail I'd
like to know more about ansible
19:05:06 <nirik> cool. See me after the meeting in #fedora-admin and I can make sure
you are added to our apprentice sysadmin group and pointed at tickets and such. ;)
19:05:13 <nirik> and welcome again!
19:05:20 <axil42> cool :)
19:05:30 <purplex> sure! :)
19:06:05 <nirik> #topic Meeting time (DST)
19:06:15 <nirik> so the us goes to daylight savings time next week...
19:06:36 <nirik> I'd like to move the meeting forward an hour to be at the same
time... so 20UTC... is that ok for everyone?
19:06:54 <nirik> (at least if i got that right, I hate timezones and dst)
19:07:01 <purplex> yes the time will be alright.
19:07:29 <relrod> worksforme
19:07:29 <johe> +1
19:07:38 <pingou> so for couple of weeks it's gonna be at 21:00 (9pm) instead of
20:00 (8pm) for western europ
19:07:53 <webpigeon> that's fine for me
19:08:19 <nirik> ok, cool. :)
19:08:32 <pingou> wfm :)
19:08:36 <nirik> #info meeting to move ahead an hour next week
19:08:46 <pingou> for those that changes to DST :)
19:08:52 <nirik> #topic Applications status / discussion
19:08:58 <pingou> otherwise it moves one hour back :)
19:08:59 <nirik> any applications status or news?
19:09:06 <nirik> pingou: right, true.
19:09:08 <pingou> there is a new FAS release, 0.9.0
19:09:12 <nirik> dst gives me a headache.
19:09:23 <pingou>
https://admin.stg.fedoraproject.org/accounts stg is updated
19:09:33 <pingou> pkgdb2-cli is almost ready
19:09:45 <abadger1999> yay!
19:09:48 <smooge> yay!
19:09:50 <pingou> I might still tweak a little the api for pkgdb2
19:09:51 <smooge> psych
19:10:02 <pingou> but basically I'm hoping to push pkgdb2 to stg next week
19:10:10 <abadger1999> pingou: Do we need a new python-fedora release to fgo with
that?
19:10:17 <pingou> then push it to prod, then take, I don't know, a month off?
:)
19:10:44 <pingou> abadger1999: I went the way that pkgdb-cli has the login mechanism
embedded, until we can figure out python-fedora
19:10:49 <abadger1999> <nod>
19:11:06 <pingou> so pkgdb2 can move forward :)
19:11:14 <nirik> I see 3 fas hotfixes. I'll ping SmootherFrOgZ to make sure they
are rolled into upstream.
19:11:37 <pingou> cool thanks
19:11:56 <pingou> abadger1999: if you could ping fchiulli about the pending pr on
elections2
19:12:13 <pingou> ?
19:12:17 <pingou> pretty please :)
19:12:18 <nirik> #info new fas release out soon, stg updated
19:12:25 * pingou cats eyes mode off
19:12:39 <nirik> #info pkgdb2-cli is close to ready and pkgdb2 hopefully to stg next
week
19:12:43 <pingou> note for the new fas release, there is a db change
19:13:00 <nirik> ok, so we will likely need an outage for it... for prod
19:13:04 <pingou>
https://github.com/fedora-infra/fas/blob/develop/updates/fas2-0.8.9.1_to_...
19:13:10 <abadger1999> pingou: will do.
19:13:16 <pingou> and the script name doesn't reflect the version change
19:13:19 <pingou> abadger1999: thanks :)
19:14:07 <pingou> oh and there is a fedocal 0.5.1 in stg
19:14:16 <pingou> that I'll push to prod some time soon :)
19:14:41 <nirik> cool. Just some bugfixes? or any new goodies?
19:14:57 <pingou> a little UI improvement and bugfix in the reminder email
19:15:21 <pingou>
https://apps.stg.fedoraproject.org/calendar/fedora-meeting/ check
the month calendar on the side
19:15:29 <pingou> (Oo the db has been synced from prod to stg?)
19:15:47 <nirik> nice
19:16:32 <nirik> oh...
19:16:46 <nirik> #info bodhi01 / 02 production instances added to the bodhi app
pool
19:17:03 <nirik> if everything looks ok, we will drop the app01-04 next week
19:17:10 <nirik> and another thing will be off the app servers.
19:17:25 <lmacken> yay!
19:17:29 <pingou> cool
19:17:52 <abadger1999> Cool.
19:18:26 <nirik> I'll possibly file some more tickets on other things to move
off.
19:19:26 <smooge> what is left on the app boxes?
19:19:47 <nirik> whats left: wiki, mirrormanager admin interface, pkgdb (will be
done when we move to 2), elections (will be done with the move to new), freemedia, geoip,
busmon
19:20:20 <nirik> those last 3 perhaps can go to the same pair of new ones... they
are all small.
19:21:08 <nirik> ok, anything else on the application devel side of things?
19:21:17 <willo> just quickly, geoip and freemedia are not on
puppettoansiblemigration
19:21:22 <willo> should they be?
19:21:33 <nirik> willo: yeah, we should add them... as apps I guess?
19:21:38 <willo> Will do
19:21:45 <nirik> they are both really simple... one is a wsgi and the other is a cgi
I think?
19:21:56 <nirik> #topic Sysadmin status / discussion
19:22:00 <nirik> Speaking of which...
19:22:15 <nirik> willo put together a page for us on puppet-> ansible migration.
;)
19:22:20 * lbazan here
19:22:42 <nirik>
https://fedoraproject.org/wiki/Infrastructure/PuppetToAnsibleMigration
19:22:54 <pingou> nice work :)
19:23:58 <nirik> In the last week or two I have also taken work from janeznemanic
and adimania and migrated smtp-mm servers and paste over.
19:24:09 <nirik> Thanks for all the help everyone, we are on a roll. ;)
19:24:13 <willo> wiki page updated with geoip and freemedia
19:24:42 <nirik> #info paste and smtp-mm migrated over recently
19:24:42 <willo> they weren't on that list because they weren't on app.fp.o
19:24:53 <willo> should they be there as well
19:25:36 <nirik> willo: I guess we could add freemedia... geoip isn't really
something users directly go to. It's used by anaconda to tell timezone/lang on
installs and by gnome for maps location or the like...
19:25:56 <nirik> I mean they do directly query it, but you wouldnt normally go there
in a web browser.
19:25:58 <willo> i also don't know the URL for those two apps. so, if someone
cam update the page, then that'll be awesome
19:26:49 <nirik>
https://fedoraproject.org/freemedia/FreeMedia-form.html is
freemedia
19:28:02 <nirik> lets see...
19:28:02 <willo> Ok, I'll add freemedia to apps.fp.o.
19:28:14 <nirik> #info updates applied all around yesterday/day before.
19:28:29 <nirik> we have some new machines rolling in soon... smooge: any news on
those?
19:28:33 <smooge> I think I have found a couple of 'solutions' for the
download servers
19:28:52 <nirik> smooge: I saw that email with tuning... it's worth trying, but
I think we already tried similar.
19:28:54 <smooge> the guys are stuck on another project in PHX1 so haven't had
time. I have an open ticket on the new boxes
19:29:03 <nirik> ok
19:29:24 <nirik> #info ansible 1.5 came out recently. Some nice improvements.
19:29:35 <smooge> nirik, this option is mount level and basically tells the nfs side
to actually do what the flags we set did on local disks
19:30:09 <smooge> I found another option which is for read-only mounts that
don't change a lot. I am not sure about how we fit in that
19:30:10 <nirik> smooge: ok. Do give it a whirl. Anything we can do to make it
better without impacting our tier1 mirrors is good.
19:30:57 <smooge> well at least I got something productive this morning... :)
19:31:08 <nirik> FYI, for anyone lost here... we have been having problems with the
netapp storage behind our main download servers. It's been very busy. We are trying to
make it cache better/behave nicer... while waiting for them to move us to new hardware.
19:32:04 <pingou> eta on the new hardware?
19:32:24 <nirik> hard to say. "soon" they keep telling us. ;)
19:32:48 <mirek-hm> Q: who is sysadmin capable of fixing stuff in fedora cloud in
europe timezone?
19:33:21 <smooge> no one?
19:33:45 <smooge> puiterwijk, is in europe timezone but he is not a fulltime
sysadmin
19:34:07 <mirek-hm> ok
19:34:12 <nirik> actually I think SmootherFrOgZ has access to the nodes...
19:34:30 <nirik> mirek-hm: I'd be ok with givig you ssh access to the nodes if
it would help?
19:35:03 <nirik> I really want to make a new clean cloud on the new machines we
get... migrate everything to it, then redo this cloud clean too...
19:35:11 <nirik> so it's nice and redeployable and such.
19:35:59 <smooge> yes.
19:36:11 <mirek-hm> nirik: *nod* I just want to avoid in future, what happen today
when FC was dozen hours non-functional and waiting just on nirik
19:37:09 <nirik> mirek-hm: yeah. :( I can add your key to the fed-cloud02.cloud node
(it's the controller). Do be very carefull there tho... it's older and manually
setup so if we break things it may be hard to fix. ;)
19:37:31 <smooge> s/hard/impossiblish/
19:37:31 <nirik> and sadly it's not often clear what the cause of issues is. :9
19:37:57 <smooge> oh I like that emoticon :9 when I am confused and sad
19:38:36 <nirik> I meant :), but it came out ;9
19:38:43 <pingou> :)
19:38:51 <nirik> anyhow, any other sysadmin related stuff?
19:38:58 <johe> by the way, what type of cloud software do we use?!?
19:39:05 <nirik> johe: openstack
19:39:13 <johe> makes sense :-)
19:39:59 <nirik> oh, related to both ansible and openstack/cloud: I fixed our
playbooks for temp cloud instances the other week... they should work nicely now to just
spin up something to test with.
19:40:02 <smooge> which explains why my euca notes didn't work
19:40:40 <nirik> I am happy I figured out I could make the second play run as the
'fedora' user with sudo and avoid the anoying root cannot login thing.
19:40:46 <pingou> nice
19:41:05 <nirik> #topic Upcoming Tasks/Items
19:41:05 <nirik>
https://apps.fedoraproject.org/calendar/list/infrastructure/
19:41:15 <nirik> any upcoming items folks would like to schedule or note?
19:42:06 <nirik> I did a tiny bit more tweaking on the FAD page...
19:42:16 <nirik>
https://fedoraproject.org/wiki/FAD_Bodhi2_Taskotron_2014
19:42:26 <nirik> still needs a bunch more work before we can move forward with it.
19:43:04 <danofsatx-work> just for the record, I am here - missed the roll call.
19:43:17 <nirik> hey danofsatx-work. :)
19:44:17 <nirik> #topic Open Floor
19:44:36 <nirik> Anyone have items for open floor? questions? comments? favorate
colors?
19:44:57 <smooge> I am rebooting dl05 at the moment. stupid mount
19:45:05 <nirik> fun.
19:45:33 <nirik> ok, if nothing else, will close out in a minute...
19:45:34 <smooge> so when we redo the cloud... will they be in a different ansible
tree or the main one?
19:45:46 <nirik> smooge: well, ideally they would be in our main ansible repo.
19:46:00 <nirik> but I can't find a good workable ansible playbook for making
them.
19:46:21 <nirik> so, we may end up using some other tool to install/deploy and
configure on top of that with ansible.
19:46:21 <smooge> the cloudlets ro the servers on top?
19:46:45 <nirik> installing the actual openstack cloud on the hardware.
19:46:53 <nirik> Managing it after it's up in much easier.
19:47:57 <nirik> but we can figure something out I'm sure...
19:48:20 <smooge> I was thinking we could take one fo the boxes that are under
warranty but being replaced and put them up as a lockbox -like box on the network
19:48:50 <nirik> thats not really the problem tho. We can reach them via ssh to
configure, it's just that setting up openstack is a pain in the rear.
19:49:00 <nirik> it has 50 components and databases and ...
19:49:25 <tflink> IIRC, it took me a day or so to set up my single-node openstack
system
19:49:45 <smooge> oi
19:49:47 <tflink> and that's without a distributed fs or the other components
that a multi-node system needs
19:50:11 <nirik> there's 19 services on fed-cloud02 set to start on boot with
'openstack' in their names.
19:50:27 <pingou> 'nice'
19:50:51 <danofsatx-work> what services are those?
19:50:54 <nirik> anyhow, we will get there. I might just use some other scripting
and get it initally setup, then use ansible on top
19:51:32 <nirik>
http://paste.fedoraproject.org/83130/94135484/
19:51:38 <danofsatx-work> don't need much more than keystone and nova, unless
it's a storage node
19:51:57 <nirik> it is. also network, image store, etc.
19:52:23 <nirik> anyhow... we will sort out something.
19:52:29 <danofsatx-work> oh, I see openstack is getting bit by the creep bug, too.
19:52:36 * danofsatx-work sighs
19:52:51 <nirik> danofsatx-work: it's always had a bunch of components. With
silly names. ;)
20:09:35 <nirik> #endmeeting