#fedora-meeting: Infrastructure (2013-02-21)
Meeting started by nirik at 19:00:01 UTC. The full logs are available at
* welcome y'all (nirik, 19:00:01)
* New folks introductions and Apprentice tasks. (nirik, 19:02:15)
* new easyfix tasks welcome, team members are encouraged to try and
file tickets for them. (nirik, 19:05:28)
* Applications status / discussion (nirik, 19:06:17)
* pingou has vastly simplified the pkgdb db. (nirik, 19:07:42)
* new pkgdb-cli pushed out as well as copr-cli (nirik, 19:08:16)
* fas release being tested in staging, for 2013-02-28 release to prod.
* askbot is now sending fedmsg's. (nirik, 19:11:56)
* more fas-openid testing welcome. Has worked for those folks that
have tried it so far. (nirik, 19:15:29)
* fedocal ready for 1.0 tag and review process. (nirik, 19:16:16)
* LINK: http://elections-dev.cloud.fedoraproject.org/
* testing on new elections version welcome:
(make account in
fakefas) (nirik, 19:17:04)
* will try out an f18 server for mm3 staging testing and feel out an
updates policy, etc. Possibly using snapshots more. (nirik,
* will look at moving fas-openid to prod as soon as is feasable.
* feedback on github reviews of all commits welcome. (nirik,
* mirrormanager update to 1.4 soon. (nirik, 19:39:11)
* Sysadmin status / discussion (nirik, 19:43:00)
* smooge got our bnfs01 server's disks working again. (nirik,
* nagios adjustments in progress (nirik, 19:44:30)
* arm boxes will get new net friday hopefully (nirik, 19:45:07)
* mass reboot next wed (tenative) for rhel 6.4 upgrades. (nirik,
* Private Cloud status update / discussion (nirik, 19:52:50)
* euca cloudlet limping along after upgrade. (nirik, 19:55:11)
* work on going to bring openstack cloudlet up to more production
* please see skvidal if you want to get involved in our private cloud
setup (nirik, 20:01:29)
* Upcoming Tasks/Items (nirik, 20:01:33)
* 2013-02-28 end of 4th quarter (nirik, 20:01:44)
* 2013-03-01 nag fi-apprentices (nirik, 20:01:44)
* 2013-03-07 remove inactive apprentices. (nirik, 20:01:44)
* 2013-03-19 to 2013-03-26 - koji update (nirik, 20:01:44)
* 2013-03-29 - spring holiday. (nirik, 20:01:44)
* 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze (nirik,
* 2013-04-16 F19 alpha release (nirik, 20:01:48)
* 2013-05-07 to 2013-05-21 BETA infrastructure freeze (nirik,
* 2013-05-21 F19 beta release (nirik, 20:01:52)
* 2013-05-31 end of 1st quarter (nirik, 20:01:54)
* 2013-06-11 to 2013-06-25 FINAL infrastructure freeze. (nirik,
* 2013-06-25 F19 FINAL release (nirik, 20:01:58)
* Open Floor (nirik, 20:02:49)
Meeting ended at 20:04:14 UTC.
Action Items, by person
People Present (lines said)
* nirik (143)
* skvidal (99)
* abadger1999 (47)
* pingou (24)
* abompard (15)
* smooge (10)
* mdomsch (10)
* threebean (6)
* zodbot (5)
* SmootherFrOgZ (4)
* cyberworm54 (4)
* lmacken (2)
* maayke (1)
* ricky (0)
* dgilmore (0)
* CodeBlock (0)
19:00:01 <nirik> #startmeeting Infrastructure (2013-02-21)
19:00:01 <zodbot> Meeting started Thu Feb 21 19:00:01 2013 UTC. The chair is nirik.
Information about MeetBot at http://wiki.debian.org/MeetBot
19:00:01 <zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
19:00:01 <nirik> #meetingname infrastructure
19:00:01 <zodbot> The meeting name has been set to 'infrastructure'
19:00:01 <nirik> #topic welcome y'all
19:00:01 <nirik> #chair smooge skvidal CodeBlock ricky nirik abadger1999 lmacken
dgilmore mdomsch threebean
19:00:01 <zodbot> Current chairs: CodeBlock abadger1999 dgilmore lmacken mdomsch
nirik ricky skvidal smooge threebean
19:00:13 * skvidal is here
19:00:15 <nirik> hello everyone. whos around for an infrastructure meeting?
19:00:15 <smooge> not guilty
19:00:23 * cyberworm54 is here
19:00:25 * lmacken
19:00:26 * threebean is kinda here
19:00:28 * maayke is here
19:00:33 * abadger1999 here
19:00:40 * pingou here
19:00:52 * SmootherFrOgZ here
19:02:08 <nirik> ok, I guess lets go ahead and dive in...
19:02:15 <nirik> #topic New folks introductions and Apprentice tasks.
19:02:30 <nirik> any new folks like to introduce themselves? or apprentices with
questions or comments?
19:03:04 <cyberworm54> Hi I am an apprentice and hopefully I can learn and
contribute as much as I can
19:03:31 <nirik> welcome (back) cyberworm54
19:03:57 <cyberworm54> Thanks!
19:04:01 <nirik> to digress a bit... do folks think our apprentice setup is working
well? or is there anything we can do to improve it?
19:04:20 <nirik> I think the biggest problem is new people getting up to speed and
finding things they can work on.
19:04:52 <skvidal> nirik: also - we have a fair amount more code-related tasks than
general admin tasks that newcomers can get into
19:04:56 <nirik> we are also low on new easyfix tickets, particularly in the
19:05:02 <nirik> yeah.
19:05:14 <cyberworm54> it is a bit ...confusing but once you get to the docs and
actually read it you have a start point
19:05:28 <nirik> #info new easyfix tasks welcome, team members are encouraged to try
and file tickets for them.
19:06:06 <nirik> ok, moving on then I guess.
19:06:17 <nirik> #topic Applications status / discussion
19:06:27 <nirik> any application / development news this week or upcoming?
19:06:46 <pingou> I've been doing some cleanup on the pkgdb db scheme
19:06:49 <pingou> before: http://ambre.pingoured.fr/public/pkgdb.png
19:06:57 <pingou> after: http://ambre.pingoured.fr/public/pkgdb2.png
19:07:25 <pingou> that's with the help of abadger1999 :)
19:07:29 <nirik> wow. nice!
19:07:29 <lmacken> nice ☺
19:07:42 <nirik> #info pingou has vastly simplified the pkgdb db.
19:07:46 * abadger1999 just reviews and makes suggestions to what pingou writes ;-)
19:07:54 <pingou> pushed a new version of pkgdb-cli (waiting to arrive in testing)
and pushed upstream a new version of copr-cli
19:08:16 <nirik> #info new pkgdb-cli pushed out as well as copr-cli
19:08:19 <abadger1999> New fas release is finally out the door. Planning to upgrade
production on Feb 28.
19:08:29 <pingou> abadger1999 and I have started to think about pkgdb2 basically,
schema update is the first step
19:08:56 <abadger1999> pkgdb -- yeah, and pkgdb2 api is probably going to be the
19:08:57 <nirik> #info fas release being tested in staging, for 2013-02-28 release
19:09:19 <abadger1999> as a note for admins -- the fas release that introduced
fedmsg introduced a bug that you should know about
19:09:40 <SmootherFrOgZ> btw, there's a bunch of locale fixes in the new fas
19:09:41 <abadger1999> email verification when people change their email address was
19:09:50 <nirik> thats the one we have in prod, but we have hotfixed it right?
19:10:00 <SmootherFrOgZ> would good to test fas with different languages
19:10:32 <nirik> cool.
19:10:39 <abadger1999> it would change the email when the user first entered the
updated email in the form instead of waiting for them to confirm that the received the
19:10:45 <nirik> I saw in stg that it also has the 'no longer accept just
yubikey for password' in.
19:11:37 <threebean> askbot got fedmsg hooks in production this week. there are
some new bugs to chase down regarding invalid sigs and busted links..
19:11:41 <nirik> any other application news? oh...
19:11:56 <nirik> #info askbot is now sending fedmsg's.
19:11:58 <threebean> Latest status -> http://www.fedmsg.com/en/latest/status/
19:12:08 <skvidal> fedmsg.com? wow
19:12:25 <nirik> Has anyone had a chance to test patrick's fas-openid dev
instance? any feedback for him?
19:12:26 <abadger1999> nirik: Hmm... looks like production isn't hotfixed.
19:12:30 <skvidal> threebean: what's the status on fedmsg emitters from outside
of the vpn?
19:12:35 <abadger1999> nirik: but next fas release will have the fix.
19:12:40 <nirik> abadger1999: :( I thought we did. ok.
19:12:47 <threebean> skvidal: no material progress yet, but I've been thinking
19:12:50 <abadger1999> Can we wait until Thursday?
19:13:01 <skvidal> threebean: okay thanks
19:13:04 <threebean> skvidal: I have some janitorial work to do.. then that's
next on my list.
19:13:21 <skvidal> threebean: that's the limiting factor for adding notices from
coprs, I think
19:13:29 <nirik> abadger1999: I suppose
19:14:12 * threebean nods
19:14:18 <abadger1999> I've used fas-openid but not tested it heavily. It has
worked and looks nice. puiterwijk has a flask-fas-openid auth plugin that he's tested
and converted fedocal, IIRC, to use it.
19:14:41 <nirik> yeah, it's worked for me for a small set of sites I tested.
19:15:22 <pingou> speaking of fedocal, I need to tag 0.1.0 and put it up for review
19:15:29 <nirik> #info more fas-openid testing welcome. Has worked for those folks
that have tried it so far.
19:15:41 <pingou> the current feature requests will have to wait for the next
19:15:57 <nirik> pingou: yeah. Will be good to get it setup. :)
19:16:15 <abadger1999> Oh, fchiulli has a new version of elections that's ready
for some light testing
19:16:16 <nirik> #info fedocal ready for 1.0 tag and review process.
19:16:24 <pingou> abadger1999: oh cool!
19:16:30 <abadger1999> http://elections-dev.cloud.fedoraproject.org/
19:16:31 <nirik> abadger1999: cool. Is there an instance up?
19:16:34 <nirik> nice.
19:16:44 <skvidal> nirik: should be
19:16:48 <abadger1999> You need to make an account on fakefas in order to try it
19:17:04 <nirik> #info testing on new elections version welcome:
(make account in fakefas)
19:17:06 <abadger1999> Please do try it out.
19:17:06 <skvidal> abadger1999: is elections switching to fas-openid, too?
19:17:24 <pingou> abadger1999: and the code is ?
19:17:45 <abadger1999> skvidal: I believe it is using flask-fas right now because
flask-fas-openid isn't in a released python-fedora yet.
19:18:03 <abadger1999> pingou: https://github.com/fedora-infra/elections
19:18:14 <skvidal> abadger1999: got it
19:18:19 <pingou> abadger1999: great
19:18:20 <skvidal> abadger1999: thx
19:18:36 <abadger1999> np
19:18:47 <nirik> I have one more application type thing to discuss... dunno if
abompard is still awake, but we should discuss mailman3. ;)
19:18:51 <abadger1999> I am all for moving more things over to the flask-fas-openid
19:19:15 * nirik is too.
19:19:33 <nirik> anyhow, we are looking at setting up a mailman3 staging to do some
more testing and shake things out.
19:19:41 <nirik> however, mailman3 needs python 2.7
19:19:43 <abompard> nirik: yeaj
19:20:06 <nirik> so, it seems: a) rhel6 + a bunch of python rpms we build and
maintain against python 2.7
19:20:12 <nirik> or b) fedora 18 instance
19:20:30 <smooge> abadger1999, congrats on election stuff
19:20:38 <abompard> yes, and MM3 really does not work on python 2.6, sadly
19:20:47 * pingou question: which one will be out first: EL7 or MM3? :-p
19:20:55 <nirik> we are starting to have more fedora in our infra (for example, the
arm builders are all f18)
19:21:09 <nirik> so, we might want to come up with some policy/process around them.
Like when do to updates, etc.
19:21:09 <abadger1999> smooge: thanks. It was all fchiulli though :-) I told him
he can be the new owner of the code too :-)
19:21:13 <abompard> I've already rebuilt an application for a non-system python,
and it's not much fun
19:21:33 <smooge> bwahahahah
19:21:33 <abompard> as in non-scriptable
19:21:58 <nirik> yeah, it's pain either way...
19:21:59 * abadger1999 thinks fedora boxes are going to be preferable to non-system
19:22:07 <pingou> +1
19:22:09 <skvidal> nirik: an idea
19:22:11 <nirik> I'm leaning that way as well.
19:22:16 <abompard> by the way, Debian has a strange but nifty packaging policy for
python package that make them work with all the installed versions of python
19:22:21 <smooge> I think we should make a bunch of servers rawhide
19:22:40 <skvidal> abompard: I assume the db /data for mm3 is all separate from
where it needs to run, right
19:22:46 <abadger1999> abompard: yeah -- I've looked at the policy but not hte
implementation. But every time I've run it by dmalcolm, he's said he doesn't
19:23:04 <abadger1999> abompard: i think some of that might be because he has looked
at the implementation :-)
19:23:05 <abompard> abadger1999: understandably, it's symlink-based
19:23:17 <abompard> skvidal: yeah, to some extent
19:23:23 <skvidal> nirik: I wonder if we could have 2 instances - talking to the
same db - so we could update f18 to latest - run mm3 on it in r/o mode - to make sure it
19:23:27 <abompard> skvidal: it has local spool directories
19:23:30 <skvidal> nirik: then just pass the ip over to the other one
19:23:40 <nirik> in the past we have been shy of fedora instances because of the
massive updates flow I think, as well as possible bugs around those updates. I think
it's gotten much better in the last few years (I like to think due to the updates
policy, but hard to say)
19:23:59 <skvidal> nirik: which is why I was thinking we don't do updates to the
19:24:07 <skvidal> we just swap out the instance that is in use/has that ip
19:24:08 <abadger1999> ... or less contributors? /me ducks and runs
19:24:16 <nirik> :)
19:24:22 <skvidal> nirik: so we test the 'install'
19:24:24 <nirik> skvidal: right, so a extra level of staging?
19:24:31 <skvidal> nirik: one level, really
19:24:32 <abompard> skvidal: I don't know how MM3 will handle a read-only DB
19:24:37 <skvidal> prod and staging
19:25:02 <nirik> well, right now we are talking about a staging instance only, but
yeah, I see what you mean. we could do something along those lines.
19:25:17 <nirik> I also think for some use cases it's not as likely to break...
19:25:36 <nirik> ie, for mailman, postfix and mailman and httpd all need to work,
but it doesn't need super leaf nodes right?
19:25:39 <skvidal> abompard: understood
19:26:02 <skvidal> nirik: anyway - just an idea
19:26:04 <skvidal> nirik: ooo - actually
19:26:13 <skvidal> nirik: I just had a second idea that you will either hate or
19:26:14 <nirik> where as for something like a pyramid app, it would be a much more
19:26:16 <skvidal> nirik: snapshots
19:26:16 <abompard> skvidal: we may get bugs because of that, not because of the
19:26:30 <skvidal> nirik: we snapshot the running instance in the cloud
19:26:32 <nirik> yeah, we could do that too.
19:26:32 <skvidal> nirik: upgrade it
19:26:36 <skvidal> and if it dies - roll it out
19:27:04 <abompard> for the moment it will only be low-traffic lists anyway
19:27:22 <abompard> and I must check that, but if MM is not running, I think postfix
keeps the message
19:27:30 <abadger1999> skvidal: how would that work in terms of data? would we keep
the db and local spool directory separate from the snapshots?
19:27:33 <abompard> and re-delivers when MM starts
19:27:34 <skvidal> abompard: yes
19:27:35 <nirik> FWIW, I run f18 servers at home here, and they have been pretty
darn stable. (as they were when f17... earlier releases had more breakage from my
19:27:41 <skvidal> err
19:27:41 <skvidal> abadger1999: yes
19:27:44 <abadger1999> Cool.
19:28:11 <skvidal> abadger1999: no reason we can't have a mm3-db server in the
19:28:12 * abadger1999 kinda likes that. although possibly he just doesn't know all
the corner cases there :-)
19:28:16 <nirik> yeah. I'm sure we could do something with snapshots.
19:28:21 <skvidal> anyway - just an idea
19:28:23 <skvidal> nothing in stone
19:28:27 <nirik> yeah.
19:29:06 <nirik> also, for updates, we may just do them on the same schedule as rhel
ones, unless something security comes up in an exposed part... ie, just look at the httpd,
etc not the entire machine.
19:29:42 <nirik> anyhow, all to be determined, we can feel out a policy.
19:29:49 <nirik> anything else on the applications side?
19:29:56 <abadger1999> I have two more
19:30:00 <abadger1999> Do we have a schedule for getting fas-openid into
19:30:28 <nirik> abadger1999: I think it's ready for stg for sure now... but not
sure when prod...
19:30:58 <nirik> I'm fine with rolling it out as fast as we are comfortable
19:31:03 <nirik> I'd like to see it get more use. ;)
19:31:04 <abadger1999> I think we're coming along great. But if we're going
to start migrating apps to use fas-openid/telling people to use it when developing their
apps (like elections), then we need to have a plan for getting it into prod
19:31:09 <abadger1999> <nod>
19:31:19 <abadger1999> nirik: it's setup to replace the current fas urls?
19:31:34 <nirik> abadger1999: not fully sure on that. I think so...
19:31:36 * abadger1999 was wondering if we could deploy it and just not announce it for a
19:31:46 <nirik> thats a thought.
19:32:22 <abadger1999> alright -- I guess let's talk about htis more on Friday
after our classroom session with puiterwijk :-)
19:32:26 <nirik> Oddly I have noticed that for things like askbot you get two
different "users" with different urls.
19:32:28 <nirik> yeah
19:33:05 <abadger1999> Other thing is for all the devs here, how's the
"review all changes" idea working out?
19:33:27 <nirik> #info will try out an f18 server for mm3 staging testing and feel
out an updates policy, etc. Possibly using snapshots more.
19:33:39 <abadger1999> I've liked how it works with pingou, puiterwijk, and
SmootherFrogZ for fas, python-fedora, and packagedb.
19:33:46 <nirik> #info will look at moving fas-openid to prod as soon as is
19:33:55 <skvidal> abompard: how much space do you need on the mm server itself - if
you are not storing the db there?
19:33:59 <abadger1999> lmacken: Is it working okay for bodhi and such too?
19:34:07 <abadger1999> anything that's falling through the cracks?
19:34:14 <abompard> skvidal: I need to check that
19:34:21 <nirik> skvidal: if we are doing this as a real staging, we might want to
just make a real 'lists01.stg.phx2' virthost instead of cloud?
19:34:26 <pingou> abompard: I defintevely like it
19:34:53 <abadger1999> Do we want to say that certain things are okay to push
without review? (making a release would be a candidate...I was going to suggest
documentation earlier but pingou found a number of problems with my documentation patch
19:34:53 <pingou> abadger1999: ^ :)
19:34:54 <skvidal> nirik: okay - I didn't know if we wanted to be cloud-er-fic
about it or not
19:35:01 <skvidal> nirik: thx
19:35:31 <nirik> skvidal: yeah, I'm open to either, but I think right now until
we have less fog in our clouds, a real one might be better for this... but either way
19:35:53 <skvidal> nirik: well - with attached persistent volumes - using one of the
qcow imgs is non-harmful
19:35:55 <nirik> abadger1999: I like seeing the extra review. I've not done much
reviewing myself. ;)
19:36:06 <abompard> skvidal: not much, a few hudred MB
19:36:08 <skvidal> nirik: but I agree about fog
19:36:17 * abadger1999 notes that threebean is in another meeting but said he still likes
the idea but hasn't done it consisstently all the time. So more experimentation with
19:36:43 * abadger1999 liked that nb reviewed a documentation update the other day :-)
19:37:02 <pingou> I think it can bring us new contributor
19:37:21 <pingou> some of them are easyfix
19:37:31 <pingou> other are bigger and then might need more experienced reviewers
19:37:57 <nirik> yeah
19:38:21 <nirik> welcome mdomsch
19:38:41 <abadger1999> Yeah. I agree. it's nice to have someone else's
eyes on the bigger fixes even if they're relatively new too, though. It's better
than before where I would have committed it without any review at all.
19:38:49 <mdomsch> better late than never
19:38:51 <nirik> that reminds me, mdomsch was going to look at updating mm in prod
to 1.4 on friday... if not then, then sometime soon. ;)
19:39:04 <nirik> #info feedback on github reviews of all commits welcome.
19:39:11 <mdomsch> anyone have any grief with doing a major MM upgrade tomorrow
19:39:11 <nirik> #info mirrormanager update to 1.4 soon.
19:39:47 <abadger1999> mdomsch: If you're around in case it goes sideways it
would be very nice.
19:39:51 <mdomsch> everything I know I've broken, I've fixed. Now it's
time to test in production. :-)
19:39:52 <nirik> I think it should be fine. We can be somewhat paranoid and not
touch one of the apps so we have an easy fallback.
19:40:11 <abadger1999> get the fixes in that you've had pending and get us onto
a single codebase for development.
19:40:13 <mdomsch> k
19:40:25 <nirik> (until we are sure the others are all working right I mean)
19:40:31 <mdomsch> right
19:40:34 <mdomsch> so bapp02, then app01
19:40:47 * nirik nods.
19:40:48 <mdomsch> and I'll stop the automatic push from bapp02 to app*
19:40:58 <nirik> sounds good.
19:41:00 <mdomsch> until we're comfortable. Worst case, we have slightly stale
data for a few hours
19:41:21 * nirik nods.
19:41:28 <abadger1999> instead of "if you're around" it would'vs
been clearer for me to say "as long as you're around" :-)
19:41:43 <nirik> mdomsch: you've picked up all the hotfixes into 1.4 right?
19:41:46 <mdomsch> abadger1999: naturally; I'm not around nearly as much
19:41:56 <abadger1999> Yeah. we miss you ;-)
19:42:16 <nirik> abadger1999: +1 :)
19:42:40 <nirik> anyhow, any other application news? or shall we move on?
19:43:00 <nirik> #topic Sysadmin status / discussion
19:43:06 <mdomsch> nirik: yes I pulled them all in while at FUDCon
19:43:17 <nirik> lets see... this week smooge was out at phx2 for a whirlwind tour.
19:43:22 <nirik> mdomsch: cool.
19:43:45 <nirik> #info smooge got out bnfs01 server's disks working again.
19:43:51 <nirik> #undo
19:43:51 <zodbot> Removing item from minutes: <MeetBot.items.Info object at
19:43:56 <nirik> #info smooge got our bnfs01 server's disks working again.
19:44:09 <smooge> kind of sort of
19:44:19 <nirik> I've been tweaking nagios of late... hopefully making it
19:44:30 <nirik> #info nagios adjustments in progress
19:44:56 <nirik> We should have net for the rest of the arm boxes friday.
19:45:07 <nirik> #info arm boxes will get new net friday hopefully
19:45:14 <skvidal> I had a discussion with the author of pynag this morning
19:45:49 <nirik> cool. Worth using for a tool for us to runtime manage nagios?
19:45:50 <skvidal> if we have people willing to spend some time - we could easily
build a query tool/cli-tool for nagios downtimes/acknowledgements/etc
19:46:07 <nirik> that would be quite handy, IMHO
19:46:12 <skvidal> nirik: it needs some code to make it work - but I think the basic
functionality is available
19:46:41 <nirik> for some things the ansible nagios module would do, but for others
it would be nice to have a command line.
19:47:15 <nirik> I'd like to look at doing a mass reboot next wed or so...
upgrade everything to rhel 6.4.
19:47:17 <SmootherFrOgZ> skvidal: interesting!
19:47:37 <nirik> Might do staging today/tomorrow to let it soak there and see if any
of our stuff breaks. ;)
19:47:52 <nirik> #info mass reboot next wed (tenative) for rhel 6.4 upgrades.
19:47:59 <skvidal> nirik: right - I'd like to be able to enhance the ansible
nagios module to be more idempotent and 'proper'
19:48:04 <skvidal> nirik: pynag _could_ do that
19:48:17 <nirik> yeah, it looks very bare bones right now.
19:48:35 <nirik> in particular we could use a 'downtime for host and all
dependent hosts' type thing
19:48:52 <skvidal> nirik: we could also use a 'give me the state of this
19:48:58 <skvidal> without having to go to the webpage
19:49:14 <skvidal> according to palli (a pynag developer) it can read status.dat
19:49:15 <skvidal> from nagios
19:49:18 <smooge> I am looking at lldpd for our PHX2 systems
Mainly to better get an idea of where things are
19:49:20 <skvidal> to determine ACTUAL state
19:49:26 <nirik> finally in the sysamin world, I'd really like to poke ansible
more and get it to where we can use it for more hosts. Keep getting sidetracked, but it
will happen! :)
19:51:04 <nirik> smooge: another thing we could look at there is
(it uses lldpd type stuff).
They are about to have their first release... so very early days.
19:51:33 <smooge> ah cool
19:51:38 <smooge> wiill look at that also
19:51:55 <nirik> oh, on nagios, I set an option: soft_state_dependencies=1
19:52:22 <nirik> this hopefully will help us not get the flurry of notices when a
machine is dropping on and off the net, or has too high a load to answer, then answers
19:52:50 <nirik> #topic Private Cloud status update / discussion
19:53:01 <nirik> skvidal: want to share your pain where we are with cloudlets? :)
19:53:08 <skvidal> sure
19:53:23 <skvidal> last week I did the euca upgrade and the wheels came right off
19:53:29 <skvidal> and then it plunged over a cliff
19:53:31 <skvidal> into a volcano
19:53:41 <pingou> sounds like a lot of fun
19:53:42 <skvidal> where it was eaten by a volcano monster
19:53:54 <smooge> who was riding a yak
19:53:56 <skvidal> anyway the euca instance is limping along at the moment with
not-occasional failures :(
19:54:04 <skvidal> smooge: and the yak had to be shaven
19:54:17 <pingou> brough back some pictures >
19:54:19 <pingou> ?
19:54:21 <skvidal> so...
19:54:35 <skvidal> I've been working on porting our imgs/amis/etc over to
19:54:44 <skvidal> and getting things more production-y in the openstack instance -
19:54:58 <skvidal> I got ssl working around the ec2 api for openstack
19:55:11 <nirik> #info euca cloudlet limping along after upgrade.
19:55:12 <skvidal> working on ssl'ing the other items
19:55:18 <skvidal> for the past couple of days
19:55:26 <nirik> #info work on going to bring openstack cloudlet up to more
19:55:28 <skvidal> I've been in a fist fight with openstack and qcow images
19:55:33 <skvidal> and resizing disks
19:55:47 <skvidal> I just got confirmation from someone that what we want to do is
just not possible at the moment :)
19:55:54 <nirik> lovely. ;(
19:56:10 <skvidal> nirik: not until we get the initramdisk to resize the partitions
19:56:17 <skvidal> so - I'm punting on this
19:56:24 <skvidal> I just put in a new ami and kernel/ramdisk combo
19:56:29 <skvidal> that's rhel6.4 latest
19:56:30 <smooge> sometimes that is best
19:56:35 <nirik> yeah. I think that could work, but needs some time to get working
right. Hopefully by the cloud-utils maintainer. ;)
19:56:38 <skvidal> and since it is an AMI it resizes the disks
19:56:50 <skvidal> what it DOES NOT DO is follow the kernel on the disk - it uses
the one(s) in the cloud
19:56:54 <skvidal> which is suck
19:57:00 <skvidal> but at least it is known/obvious suck
19:57:08 <nirik> but it should also get us moving past it for now.
19:57:11 <skvidal> I've also just built a new qcow from rhel6.4
19:57:27 <skvidal> so for systems that don't need to be on-the-fly made - we can
spin them up
19:57:31 <skvidal> growpart the partition
19:57:33 <skvidal> reboot
19:57:35 <skvidal> resize
19:57:37 <skvidal> and go
19:57:47 <skvidal> and i'm working on a playbook to handle all of the above for
19:57:51 <skvidal> and, yes, it makes me cry inside
19:58:08 <nirik> ;(
19:58:12 <skvidal> that's where we are at the moment
19:58:26 <skvidal> I am making new keys/accounts/tenants/whatever
19:58:35 <skvidal> for our lockbox 'admin' user
19:58:40 <skvidal> for making persistent instances
19:58:53 * nirik nods.
19:58:57 <skvidal> the next step is to start making use of the resource tags in
19:59:02 <skvidal> so we can more easily track all this shit
19:59:15 <skvidal> also I have to make a bunch of volumes and rsycn over all the
data from the euca volumes :(
19:59:30 <skvidal> I fully expect that last part to be a giant example of suffering
19:59:46 <nirik> yeah. we should probibly move one set of instances first and sort
out if there's any doom
19:59:57 <skvidal> if I sound kinda 'bleah' there's a reason
20:00:02 <skvidal> nirik: I thought I'd start with the fartboard
20:00:07 <nirik> heh. ok
20:00:37 <skvidal> nirik: also - now that we have instance tags - it should be
doable to write a simple 'start me up' script using ansible to spin out the
20:00:40 <skvidal> and KNOW where they are
20:00:48 <nirik> ok, we are running over time... let me quickly do upcoming and open
20:00:52 <skvidal> sorry
20:00:55 <skvidal> thx
20:00:57 <nirik> thats fine. ;) all good info
20:01:04 <skvidal> one last thing
20:01:08 <skvidal> if anyone wants to get involved
20:01:08 <skvidal> ping me
20:01:29 <nirik> #info please see skvidal if you want to get involved in our private
20:01:33 <nirik> #topic Upcoming Tasks/Items
20:01:42 <nirik> (big paste)
20:01:44 <nirik> #info 2013-02-28 end of 4th quarter
20:01:44 <nirik> #info 2013-03-01 nag fi-apprentices
20:01:44 <nirik> #info 2013-03-07 remove inactive apprentices.
20:01:44 <nirik> #info 2013-03-19 to 2013-03-26 - koji update
20:01:44 <nirik> #info 2013-03-29 - spring holiday.
20:01:46 <nirik> #info 2013-04-02 to 2013-04-16 ALPHA infrastructure freeze
20:01:48 <nirik> #info 2013-04-16 F19 alpha release
20:01:50 <nirik> #info 2013-05-07 to 2013-05-21 BETA infrastructure freeze
20:01:52 <nirik> #info 2013-05-21 F19 beta release
20:01:54 <nirik> #info 2013-05-31 end of 1st quarter
20:01:56 <nirik> #info 2013-06-11 to 2013-06-25 FINAL infrastructure freeze.
20:01:58 <nirik> #info 2013-06-25 F19 FINAL release
20:02:00 <nirik> anything people want to schedule/note etc?
20:02:07 <nirik> I'll add the fas update and the mass reboot.
20:02:20 <abadger1999> Sounds good.
20:02:49 <nirik> #topic Open Floor
20:02:54 <nirik> Anyone have items for open floor?
20:03:32 <pingou> I have a series of blog post 'Fedora-Infra: Did you know?'
coming, like once a week for the coming 4 weeks
20:03:32 <nirik> ok.
20:03:42 <skvidal> pingou: wow
20:03:46 <nirik> pingou: awesome. More blog posts would be great.
20:03:49 <pingou> short stuff, speaking about some cool features/ideas
20:03:52 <skvidal> pingou: looking forward to seeing those
20:04:10 <nirik> Thanks for coming everyone. Do continue over on our regular
20:04:14 <nirik> #endmeeting