So I'd like to put together an action plan to deal with the koji01
reboot issues. Right now, we're not capturing crash dumps on this
machine (or any other, but I'm not sure there's value in doing so
unless we have an active, systemic problem like we're facing here) -
not saying that there'd be any *to* capture, but there probably are.
I'd like to setup kdump on this machine after the beta freeze is over,
but I'd like buyin from other people before doing it. Here's what I'd
1) Present another LV from bxen02 to koji01 and mount it at /var/crash
(the rootfs on koji01 is only 10GB, and we'd need more for a crash
dump or two - I'd say 20GB would be sufficient to hold two crashes,
since it's an 8GB domain). It looks like VolGroup01 where koji01 lives
has about 80G free.
2) Install kexec-tools and configure appropriately (includes adding
crashkernel=128M@16M to grub.conf)
3) Reboot machine, and wait for it to crash again.
4) Analyze the (hopefully) resulting crash dumps :)
20:00 < mmcgrath> #startmeeting Infrastructure
20:00 < zodbot> Meeting started Thu Apr 1 20:00:34 2010 UTC. The chair is mmcgrath. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:00 < zodbot> Useful Commands: #action #agreed #halp #info #idea #link #topic.
20:00 -!- zodbot changed the topic of #fedora-meeting to: (Meeting topic: Infrastructure)
20:00 < mmcgrath> #topic Who's here?
20:00 -!- zodbot changed the topic of #fedora-meeting to: Who's here? (Meeting topic: Infrastructure)
20:00 * nirik is lurking around.
20:01 * sijis is sorta here
20:01 -!- MrTom [~quassel@fedora/MrTom] has quit Remote host closed the connection
20:01 < mmcgrath> Ok
20:02 < mmcgrath> #topic Fedora 13 Beta
20:02 -!- zodbot changed the topic of #fedora-meeting to: Fedora 13 Beta (Meeting topic: Infrastructure)
20:02 < smooge> here
20:02 < smooge> is waiting for Lenova to get a support tech
20:02 < mmcgrath> #link https://fedorahosted.org/fedora-infrastructure/report/9
20:02 < mmcgrath> So as you saw from the list, we slipped by a week.
20:02 * ricky
20:03 < mmcgrath> so the change freeze also slips by one week.
20:03 < mmcgrath> .ticket 2058
20:03 < zodbot> mmcgrath: #2058 (Verify Mirror Space) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2058
20:03 < mmcgrath> I closed that just yesterday, we're in good shape on the mirrors
20:03 < mmcgrath> .ticket 2059
20:03 < zodbot> mmcgrath: #2059 (Release Day ticket) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2059
20:03 < mmcgrath> Just a tracking ticket
20:03 < mmcgrath> .tiny 2060
20:03 < zodbot> mmcgrath: Error: '2060' is not a valid url.
20:03 < mmcgrath> .ticket 2060
20:03 < zodbot> mmcgrath: #2060 (Verify releng permissions) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2060
20:03 < mmcgrath> >:-|
20:03 < mmcgrath> smooge has that one
20:03 < mmcgrath> .ticket 2061
20:03 < zodbot> mmcgrath: #2061 (MM redirects) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2061
20:03 < mmcgrath> No mdomsch again, I wonder if the time change has made this slot bad for him.
20:03 < mmcgrath> smooge: do you want to take that one for now?
20:04 < mmcgrath> I think you may have done that for the Alpha
20:04 < ricky> I thought the redirects were automatic now or something
20:04 < mmcgrath> ricky: I think they are but mdomsch seemed to want to verify that so I'm not sure if it's done and we don't have to worry about it, or if it's an ongoing thing
20:04 < mmcgrath> we should check to see if that can be removed from our list altogether
20:05 < ricky> OK
20:05 < ricky> Here's the comment from mdomsch about it: https://fedorahosted.org/fedora-infrastructure/ticket/1993
20:06 < mmcgrath> so in theory if the bug is fixed, we don't have to worry about it?
20:06 * mmcgrath will make note to talk to sr. domsch next time he's on.
20:06 < mmcgrath> ok
20:06 < smooge> ok
20:06 < mmcgrath> .ticket 2062
20:06 < zodbot> mmcgrath: #2062 (Infrastructure Change Freeze) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2062
20:06 -!- RodrigoPadula [~rodrigopa(a)188.8.131.52] has quit Quit: Saindo
20:06 < mmcgrath> This one's in effect already
20:06 < smooge> sorry I lost net connection
20:06 < mmcgrath> .ticket 2063
20:06 < zodbot> mmcgrath: #2063 (Lessons Learned) - Fedora Infrastructure - Trac - https://fedorahosted.org/fedora-infrastructure/ticket/2063
20:06 < mmcgrath> smooge: now orries
20:06 < mmcgrath> And lessons learned isn't until after the release.
20:06 < mmcgrath> So that's the release stuff
20:06 * mmcgrath notes it's only 6 minutes since th emeeting started
20:06 < mmcgrath> that could be a new record for release tickets.
20:07 < mmcgrath> anyone have any questions or concerns about the release?
20:07 < mmcgrath> ricky: do you think websites is in good shape?
20:07 < mmcgrath> sijis: ^^
20:08 -!- J5 [~johnp(a)c-65-96-175-112.hsd1.ma.comcast.net] has quit Ping timeout: 265 seconds
20:08 < sijis> yeah. just repo stuff.
20:08 < ricky> Yup, just need to get the banners ready this weekend
20:08 < sijis> tryign to finish up get-* pages..
20:08 < sijis> which can be seen in https://stg.fp.o
20:08 < mmcgrath> sijis: how'd that go for you?
20:08 < ricky> sijis: Mind sending a link to websites-list? Looking good
20:08 < mmcgrath> the staging environment I mean.
20:09 < sijis> ricky: we need to chat about banner that too. i think tyhey are trying something different.
20:09 < sijis> mmcgrath: yeah, it was easy to setup. thanks
20:09 < sijis> ricky: will send to list
20:09 < smooge> mmcgrath, you need me to take 2061 correct?
20:10 < mmcgrath> smooge: yeah, we're pretty sure it's done already, just needs a verification
20:10 < mmcgrath> Ok, well with that we'll move on.
20:10 < mmcgrath> #topic Search Engines
20:10 -!- zodbot changed the topic of #fedora-meeting to: Search Engines (Meeting topic: Infrastructure)
20:10 < smooge> okie dokie
20:10 < mmcgrath> Doah, no a-k
20:10 < mmcgrath> I thought I saw him earlier. We'll skip this.
20:11 < mmcgrath> #topic Monitoring
20:11 -!- zodbot changed the topic of #fedora-meeting to: Monitoring (Meeting topic: Infrastructure)
20:11 < mmcgrath> I've been poking around and changing some basic behaviors in nagios.
20:11 < mmcgrath> Stuff like flagging a host as dead if nrpe is down, and not going off of pings.
20:11 < mmcgrath> I'm presently (as in just before the meeting started) trying to get some decent servergroup views in order.
20:12 < mmcgrath> just lots of general config management and things
20:12 < mmcgrath> Anyone have any questions or comments on that?
20:12 < mmcgrath> I'm disappointed that we don't have a combined trend and monitoring solution
20:12 < mmcgrath> but I have to say, collectd has been awesome.
20:12 < mmcgrath> we've learned several things about the environment just by using it
20:12 < sijis> so collectd = trend?
20:12 < smooge> I like collectd too
20:13 < mmcgrath> sijis: yeah, have you seen it?
20:13 -!- skvidal [~skvidal(a)sethost.sethdot.org] has joined #fedora-meeting
20:13 < smooge> I can see why so many comapnies wrap nagios + something like it into an overall product
20:13 < mmcgrath> it's been interesting.
20:13 < sijis> yeah, right after you noticed some db issues on smolts, i think
20:13 < mmcgrath> https://admin.fedoraproject.org/collectd/bin/index.cgi?hostname=memcached...
20:13 < mmcgrath> even stuff like that has been helpful
20:13 < mmcgrath> we've had a pretty solid 98% hit ratio in memcached.
20:14 < mmcgrath> and, surprisingly to me, all of our memcached stats fit in les then 90M of space.
20:14 < mmcgrath> I figured it would be much higher then that
20:14 < mmcgrath> but anyway, yeah. Monitoring.
20:14 < mmcgrath> any questions or comments or suggestions about our monitoring systems?
20:15 < mmcgrath> I really want us to get smarter alerts.
20:15 < mmcgrath> right now when the fit hits the shan, everything kind of goes nuts and that's just not helpful.
20:15 < mmcgrath> I also want to start taking more metrics like MTTF and MTTR for our apps.
20:15 < mmcgrath> which is kind of tricky to do at the moment because of all of our entry points.
20:16 < mmcgrath> Ok, I'll assume nothing there?
20:16 < mmcgrath> #topic Blade Center
20:16 -!- zodbot changed the topic of #fedora-meeting to: Blade Center (Meeting topic: Infrastructure)
20:16 < mmcgrath> So I'm working to get a new blade center.
20:16 < smooge> whee
20:16 < mmcgrath> Due to some internal accounting bits, it's taking longer than expected. I'm still hopeful I'll be flying out to Phoenix in mid april (just a few weeks from now) to install it.
20:17 < mmcgrath> jwb: you around?
20:17 < smooge> do you need me out there?
20:17 < mmcgrath> smooge: naw, and I don't think we have cash to send you anyway :)
20:17 < smooge> I could hitch-hike?
20:17 < mmcgrath> I *hope* it will be pretty straight forward, I have that and a few other servers to install.
20:17 < mmcgrath> smooge: maybe you could just will yourself there :)
20:17 -!- mgoldmann [~mgoldmann@redhat/jboss/mgoldmann] has quit Quit: shutdown -h now
20:18 < mmcgrath> Ok, with that I'll open the floor
20:18 < mmcgrath> #topic Open Floor
20:18 -!- zodbot changed the topic of #fedora-meeting to: Open Floor (Meeting topic: Infrastructure)
20:18 < smooge> no my teleportation skills have been pretty bad lately
20:18 < mmcgrath> anyone have anything they'd like to discuss?
20:19 < smooge> I missed anything on a week delay
20:19 < skvidal> mmcgrath: if the new blade center comes in - will we have some extra machines?
20:19 < mmcgrath> skvidal: until December we might. We'll be migrating off the old BC to the new one.
20:19 < mmcgrath> then we'll be removing the old one.
20:19 < skvidal> darn
20:19 < mmcgrath> but yeah, capacity will be up, whats up?
20:20 < skvidal> just thinking about some of the koprs stuff
20:20 < skvidal> and where we're going to find the capacity for that
20:20 < skvidal> esp disk space and builders
20:20 < jwb> mmcgrath, hm?
20:20 < mmcgrath> skvidal: ah, well. I suspect it'all take a bit to get it all together but I can certainly put it in next years budget for Q1.
20:20 < mmcgrath> and we can do some trails now. We might be able to find space for all that.
20:21 < mmcgrath> the new builders are significantly beefier then the old.
20:21 < mmcgrath> jwb: hey, so. PPC stuff. Just wanted to make you aware of my intentions and see what we can do.
20:21 < smooge> mmcgrath, how much would it be to keep the old bladecenter under warranty even if slow/old
20:21 < mmcgrath> So the new blade center has two PPC blades in it for EPEL building.
20:21 < smooge> yeah!
20:21 < mmcgrath> that means all our old PPC blades will be EOLed at the end of the year. Do you happen to have any use for them outside of the colo?
20:22 < jwb> mmcgrath, why EOLed?
20:22 -!- sharkcz [~dan(a)plz1-v-4-17.static.adsl.vol.cz] has quit Quit: Ukončuji
20:22 < mmcgrath> smooge: it'd be a few thousand if they allow it. I actually extended it to be supported for this year, it's on its 4th year.
20:22 < mmcgrath> jwb: no longer supported.
20:22 < jwb> mmcgrath, the HW or ppc as a release?
20:22 < mmcgrath> the hardware
20:22 < jwb> mmcgrath, so what are you planning on doing for F12 ppc builds?
20:22 < mmcgrath> it'd probably run just fine, but since its not under warranty (and because the power drops are pretty pricy) we won't be keeping two blade centers in there at the moment.
20:22 < jwb> because F12 will still be active at the end of the year
20:23 < mmcgrath> jwb: we'll still have some non-blade builders.
20:23 < jwb> oh, ok
20:23 < mmcgrath> ppc4 still has I think 2 years left, certainly 1 year.
20:23 < jwb> mmcgrath, i can't really host the blades anywhere myself. i can ask around but needing a BC sort of makes it hard to make use of them
20:24 < mmcgrath> jwb: yeah that's what my fear was as well. Doesn't hurt to ask though. If you happen to find someone let us know.
20:24 < jwb> will do
20:25 < mmcgrath> Ok, and with that does anyone have anything else to discuss?
20:25 < smooge> not me
20:25 < mmcgrath> allllrighty!
20:25 < mmcgrath> #endmeeting
20:25 -!- zodbot changed the topic of #fedora-meeting to: Channel is used by various Fedora groups and committees for their regular meetings | Note that meetings often get logged | For questions about using Fedora please ask in #fedora | See http://fedoraproject.org/wiki/Meeting_channel for meeting schedule
20:25 < zodbot> Meeting ended Thu Apr 1 20:25:51 2010 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot .
20:25 < zodbot> Minutes: http://meetbot.fedoraproject.org/fedora-meeting/2010-04-01/fedora-meeting...
20:25 < mmcgrath> Thanks for coming everyone!
20:25 < zodbot> Minutes (text): http://meetbot.fedoraproject.org/fedora-meeting/2010-04-01/fedora-meeting...
20:25 < zodbot> Log: http://meetbot.fedoraproject.org/fedora-meeting/2010-04-01/fedora-meeting...
FYI, this means our change freeze also slips by one week.
Despite a heroic effort by developers and testers, we have not been able
to reach Beta release criteria by the time of the Go / No Go meeting.
There are still unresolved bugs and unknown test results. Because of
this we've enacted a 1 week slip of the Beta release date. This does
not mean we will be pulling in a bunch of "nice to have" updates, we
will instead be concentrating only on release blocking issues in order
to produce an RC that achieves Beta release criteria. Promotion of
builds into "stable" for F13 will continue to be extremely targeted
until an RC goes "GOLD".
Because this is the second slip in the Fedora 13 cycle, we have also
decided to bump the rest of our release dates by 1 week. The schedules
will be updated tonight and tomorrow to reflect these new dates.
Fedora -- Freedom² is a feature!
devel-announce mailing list
devel mailing list