[15:06] mmcgrath has set the subject to fedora-infrastructure meeting -- Who's here?
[15:06] xDamox: Me
[15:06] mmcgrath: PING ALL: who's here?
[15:06] G: I'm here
[15:06] mbonnet: yo
[15:06] cemc has joined the group chat (n=gimre(a)voy.narancs.net)
[15:06] G: I'm gonna have to disappear in 15-20
[15:06] jcollie: howdy!
[15:06] f13: I'm here.
[15:06] mmcgrath: abadger1999: you around?
[15:06] abadger1999: Yep.
[15:06] mmcgrath: allrighty then
[15:07] * lmacken !
[15:07] * wolfy grabs a viewing seat
[15:07] mmcgrath has set the subject to abadger1999 - Package Database -------------
[15:07] mmcgrath: abadger1999: whats the word?
[15:07] abadger1999: We have progress
[15:07] abadger1999: Script to import pkgs is written and I'm finding all sorts of problem with owners.list.
[15:08] mmcgrath: thats good at least
[15:08] mmcgrath: how bad is it?
[15:08] abadger1999: Mostly lack of EPEL owner information for EPEL branches
[15:08] lmacken: abadger1999: is there an owners API yet? If not, want some help?
[15:08] abadger1999: lmacken: I'm using nottings owners.py.
[15:09] lmacken: abadger1999: ah, where does that code live ?
[15:09] lmacken: maybe i can stuff that into bodhi in the mean time
[15:09] abadger1999: Since the pkgdb will get rid of owners.list I'm not too worried about an API.
[15:09] G: abadger1999: in that case dgilmore would be the person to poke I think
[15:09] glezos has joined the group chat (n=glezos@fedora/glezos)
[15:09] lmacken: well, bodhi needs to know who owns what
[15:09] abadger1999: cvs-int:/cvs/extras/CVSROOT/admin/owners.py
[15:09] lmacken: thanks
[15:09] mmcgrath: abadger1999: if you do get together a comprehensive list of stuff missing from EPEL send it my way.
[15:09] abadger1999: I have a few changes to what's checked in that I'll have to add later.
[15:10] abadger1999: Will do.
[15:10] abadger1999: warren is going to work on koji syncing.
[15:10] tibbs has left ("Konversation terminated!" (n=tibbs@fedora/tibbs))
[15:10] abadger1999: I'm going to do bugzilla sync and cvs acls sync this week.
[15:10] abadger1999: G (Nigel Jones) has started looking at the code and made some changes to the way it looks.
[15:11] mmcgrath: cool
[15:11] G: (Minor to start with, just getting to grips with Turbogears and how it works)
[15:11] abadger1999: So we're on track for next week or the week after.
[15:11] mmcgrath: abadger1999: anything else?
[15:11] abadger1999: That's about it.
[15:11] mmcgrath: abadger1999: the package db only contacts the database, correct? Does it need any write access to a file system like koji or cvs?
[15:12] MrBawb has joined the group chat (i=abob(a)guppy.drown.org)
[15:12] abadger1999: i think cvs acls can pull from the packagedb for now instead of pkgdb pushing to cvs-int.
[15:12] abadger1999: koji I'm not sure about. warren's looking into it.
[15:13] mmcgrath: cool.
[15:13] mmcgrath: pull is better than push for security reasons.
[15:13] mmcgrath: same reason bodhi is deployed on app5 instead of in our cluster.
[15:13] warren: you rather koji pull from packagedb?
[15:13] mbonnet: that's not really possible
[15:13] mbonnet: unless we have a cronjob that does the sync periodically
[15:13] mmcgrath: warren: if its filesystem stuff, yes. if its db stuff then no.
[15:14] warren: abadger1999, it is all db right?
[15:14] abadger1999: mbonnet: How do you currently sync owners.list?
[15:14] warren: abadger1999, see /cvs/pkgs/CVSROOT/admin/owners-sync.py
[15:14] mbonnet: I believe the script is run by hand right now
[15:14] f13: yes, by hand
[15:15] * dgilmore is here
[15:15] f13: when we make changes to owners.list we run the sync script
[15:15] f13: less than ideal, but functional
[15:15] mmcgrath: <nod>
[15:15] abadger1999: Hmm... Is it just pushing into koji's db?
[15:15] dgilmore: abadger1999: EPEL owner information is in owners.epel.list
[15:16] abadger1999: if so it can be a cronjob or we can write a callback that pushes the information from the package db to koji when it's updated.
[15:16] rdieter has left (Remote closed the connection (n=rdieter(a)sting.unl.edu))
[15:17] mmcgrath: abadger1999: we can figure it out. We should move on for the meeting though.
[15:17] f13: abadger1999: it uses the koji API to do ownership adds/changes.
[15:17] f13: doesn't talk to the db directly
[15:17] abadger1999: f13: Sounds like it won't be a problem then.
[15:17] abadger1999: mmcgrath: And won't need to write to any filesystems.
[15:17] mmcgrath: abadger1999: excellent
[15:17] mmcgrath: k, moving on
[15:18] mmcgrath has set the subject to Config Management - mmcgrath
[15:18] mmcgrath: nothing majorly new here. I'm going through and making sure our xen dom0's are setup properly.
[15:18] mmcgrath: I've also started forcing some packages to uninstall and some services not to start (cups, gpm, etc)
[15:18] mmcgrath has set the subject to VCS - jcollie
[15:18] mmcgrath: jcollie: ping?
[15:18] jcollie: yo
[15:19] mmcgrath: jcollie: Are you still playing with VCS solutions?
[15:19] jcollie: i think that the discussion last week on -devel and -infra was good
[15:19] f13: abadger1999: it needs to be made smarter, like knowing about different owners for different tags and all that, but that's just details (:
[15:19] jcollie: i just need to sit down and write up a more concrete proposal
[15:19] mmcgrath: jcollie: me too, its gotten more interest this time around then 6 mo. ago or so
[15:20] mmcgrath: jcollie: solid, make sure to get some good input from the jeremy's and jesse's in the world
[15:20] mmcgrath: k, moving next
[15:20] jcollie: i think it'll be a mix... there'll be a repository that looks a lot like we have now, but with some meta-language or -tags to pull patches out of a "exploded tree" repo
[15:20] * mdomsch joins belatedly
[15:20] mmcgrath has set the subject to Firewall System Rewrite - lmacken skvidal
[15:20] mmcgrath: mdomsch: yo
[15:21] mmcgrath: jcollie: excellent, thanks for getting that stuff together.
[15:21] mmcgrath: lmacken: ping
[15:21] mmcgrath: skvidal: ping?
[15:21] lmacken: no updates on this from my end.. have we decided to abandon pyroman ?
[15:21] mmcgrath: lmacken: Not sure, I know xDamox has some opinions on it.
[15:21] mmcgrath: xDamox: ping?
[15:21] xDamox: yo
[15:21] mmcgrath: you had some items to discuss regarding the Firewall System?
[15:21] skvidal: mmcgrath: I think we should just go with simple iptables files in /etc/sysconfig
[15:21] xDamox: Yea, I updated the template we were using and neaten it up a little
[15:21] mbonnet: question: does our firewall system have some kind of NAT/conntrack limit?
[15:22] mmcgrath: skvidal: I agree, what about boxes that have different firewall needs though?
[15:22] xDamox: I can help 100% with the iptables writing.
[15:22] skvidal: mmcgrath: that's what puppet is for
[15:22] lmacken: taking the strict & simple rule approach sounds good to me
[15:22] mbonnet: I'm wondering exceeding that limit might be the cause of the intermittent connection drops people see connecting to koji.fp.o
[15:22] skvidal: mmcgrath: distribute files out based on host
[15:22] fchiulli has joined the group chat (i=824c4010(a)gateway/web/cgi-irc/ircatwork.com/x-04ce31ce5397d4ea)
[15:22] mmcgrath: mbonnet: Both the host based and hardware firewalls can do it but only the proxy servers actually do do it now.
[15:22] mmcgrath: skvidal: ahh, yes. A puppet template would work well for that I think.
[15:23] mmcgrath: mbonnet: I'll verify that we're not rate limiting in any way on the hardware firewall.
[15:23] xDamox: mmcgrath, do we have an up to date list of services running on each box
[15:23] xDamox: and their ports?
[15:23] mmcgrath: xDamox: we're pretty close.
[15:23] mmcgrath: skvidal: do you have a link to the iptables rules you'd suggested on the list?
[15:24] skvidal: yes
[15:24] skvidal: uno momento
[15:24] skvidal: http://linux.duke.edu/~skvidal/misc/iptables-template
[15:24] xDamox: Ok. If you could give me a copy, I could do a sample firewall for some boxes maybe and have skvidal and lmacken check it over?
[15:24] lmacken: xDamox: sounds good to me
[15:24] xDamox: that good with you too skvidal ?
[15:24] skvidal: xDamox: fine - I already have those on a couple of the boxes due to the release
[15:25] skvidal: iirc they're on proxy1 and 2
[15:25] mmcgrath: xDamox: remember KISS
[15:25] xDamox: Ok, yep
[15:25] xDamox: Ill make it a simple as possible
[15:25] G: Have fun with the rest of meeting, I'm out
[15:25] mmcgrath: G: later, thanks for coming
[15:26] xDamox: also I am sure skvidal and lmacken will be able to simplify it more
[15:26] mmcgrath: xDamox: cool, take what skvidal has at http://linux.duke.edu/~skvidal/misc/iptables-template and give it a good lookover.
[15:26] mmcgrath: I'll create an erb (puppet template) out of it and see how it goes.
[15:26] xDamox: yep will do
[15:27] dgilmore: mbonnet: i dont know what on our firewalls as far as that goes
[15:27] dgilmore: mbonnet: we dont control the nat part of it
[15:28] mmcgrath: k, xDamox when you're done send'er to the list and we can get this all underway.
[15:28] mmcgrath has set the subject to Server Upgrades - mmcgrath
[15:28] xDamox: OK mmcgrath,
[15:28] mmcgrath: So I'm trying to get some additional RAM in some of our servers.
[15:28] mmcgrath: but we have more pressing issues.. .namely a lot of our newer boxes don't have warrantys.
[15:29] mmcgrath: so I'm trying to figure out where money should come from to pay for that.
[15:29] mmcgrath: additionally we have a lot of boxes that are reaching the end of their natural life and should be replaced.
[15:29] mmcgrath: Fortunately if we stick with high capacity devices, this will allow us to use our rack more efficiently.
[15:29] mmcgrath: The major limiting factor being cost, heat and power.
[15:29] mmcgrath: just letting everyone know whats going on there.
[15:30] mmcgrath has set the subject to Xen Conversion - mmcgrath
[15:30] mmcgrath: So I've started doing some work with iscsi
[15:30] mmcgrath: It's actually going quite well.
[15:30] * mmcgrath digs up a bonnie run
[15:30] warren: what will serve iscsi?
[15:30] mmcgrath: warren: the netapp already is.
[15:31] mmcgrath: grr pastebin
[15:31] dgilmore: mbonnet: how much storage do we have? how much did you use for iscsi?
[15:31] dgilmore: mmcgrath: http://paste.ausil.us
[15:32] dgilmore: mmcgrath: ^^^^^^^^^^^^^^ meant you not mbonnet
[15:32] mmcgrath: dgilmore: already on it
[15:32] mmcgrath: ok, here's an iscsi run on publictest9
[15:32] mmcgrath: http://paste.ausil.us/161
[15:32] mmcgrath: dgilmore: right now 500G
[15:32] mmcgrath: all in all I've been quite pleased with it.
[15:33] mmcgrath: I've kickstarted a few boxes with iscsi, the package install portion (about 400 packages) takes about 2 minutes.
[15:33] londo: mmcgrath: random access seems slow to me
[15:33] dgilmore: mmcgrath: live migration is easy to do?
[15:33] mmcgrath: dgilmore: yep, so far its just worked for me. There's a brief network blip I need to work on. The box itself doesn't experience it that bad but I think there's some arp issues.
[15:34] Karl_le_Rouge has joined the group chat (n=RedKarl(a)ALyon-257-1-149-122.w81-251.abo.wanadoo.fr)
[15:35] dgilmore: awesome
[15:35] mmcgrath: londo: I've seen random seek as high as 705.2.
[15:35] mmcgrath: The larger the test was on iscsi the slower that got though, always a good excuse to tweak and test though
[15:36] mmcgrath: All in all I think iscsi will work very well for us. We just need to watch carefully network utilization and overall netapp utilization.
[15:36] londo: mmcgrath: numbers from tiobench will be nice if you can get them
[15:36] mmcgrath: londo: is it in extras?
[15:36] londo: mmcgrath: yeap
[15:36] mmcgrath: londo: cool, I'll run it then.
[15:36] mmcgrath: k, moving on
[15:37] mmcgrath has set the subject to Bacula
[15:37] mmcgrath: So I've been testing out bacula on xen6 and publictest[3-4]
[15:37] mmcgrath: everything's been working great.
[15:37] f13: hurray!
[15:37] dgilmore: mmcgrath: how much total disk do we need to backup?
[15:37] mmcgrath: We're just blocking on https://bugzilla.redhat.com/230344
[15:37] f13: a scary amount
[15:37] f13: (if you count /mnt/koji)
[15:38] mmcgrath: dgilmore: wellllll, depends, do you con't /mnt/koji or not?
[15:38] dgilmore: welll we really should backup /mnt/koji
[15:38] f13: mmcgrath: btw, did the new disk shelf show up in phx?
[15:38] mmcgrath: dgilmore: the plan right now is to do a backup of everything on xen6's local storage which is 378G. I'm working on getting a tape backup for everything though (including koji)
[15:38] * dgilmore needs a cloning machine
[15:39] mmcgrath: f13: I've not heard one way or the other but I was under the impression that it should be there by now. I'll send an emil.
[15:39] dgilmore: mmcgrath: ok
[15:39] mmcgrath: dgilmore: I've got the tape drive as a priority2 thing after our warranty issue with the soc.
[15:39] mmcgrath: all in all though, ixs says he'll have more time in the comming days for us to do a formal review.
[15:40] dgilmore: mmcgrath: yeah we would probably want LT)2 or 3 with at least 10 slots
[15:40] mmcgrath: For those that haven't used it Bacula is really slick.
[15:40] skvidal: is it wicked slick?
[15:40] londo: if you are going to move things on netapp/iscsi is it possible to do the backup there (if a tape drive is available)
[15:40] mmcgrath: super wicked slick.
[15:40] abadger1999: skvidal: wykd
[15:40] dgilmore: i need to find time to get it reviewed
[15:40] mmcgrath: londo: thats the problem, we had 3 netapps to deal with now we have 1 super netapp and I'm not comfortable with backing up to itself.
[15:41] mmcgrath: londo: sorry, I missed your (if tape drive) comment.
[15:41] dgilmore: mmcgrath: i agree
[15:41] f13: I loved bacula when I was using it.
[15:41] mmcgrath: k, moving on
[15:41] mmcgrath has set the subject to Translators stuff -
[15:42] f13: seriously hot stuff
[15:42] mmcgrath: glezos: has been working on this. Its now at http://publictest4.fedora.redhat.com/
[15:42] mmcgrath: this will be a very big deal when we start moving stuff to it.
[15:42] RedKarl has left (Connection timed out (n=RedKarl(a)ALyon-257-1-39-129.w90-14.abo.wanadoo.fr))
[15:42] JSchmitt has left ("Konversation terminated!" (n=s4504kr@fedora/JSchmitt))
[15:42] mmcgrath: so all keep your eyes out for it and help out because all parties involved can use it.
[15:42] mmcgrath has set the subject to account system -
[15:42] mmcgrath: Nothing new here. If anyone is interested in working on it with me that would be good.
[15:43] mmcgrath has set the subject to Project Hosted - f13
[15:43] mmcgrath: f13: ?
[15:43] f13: nothing new. Trac git plugin sucks.
[15:44] mmcgrath: <nod>
[15:44] f13: Oh, I created a script to create trac projects, but havne't put it in scm any where or documented it
[15:44] abadger1999: f13: Could you give me access to the hosted box?
[15:44] f13: sure.
[15:44] abadger1999: Thanks.
[15:44] f13: at some point it should be FAS'd but...
[15:44] mmcgrath: <nod>
[15:45] mmcgrath: next
[15:45] mmcgrath has set the subject to FedoraPeople.org - skvidal
[15:45] mmcgrath: skvidal: anything new?
[15:45] skvidal: nothing
[15:45] warren: Is that planned for shell and web?
[15:45] skvidal: yes
[15:45] dgilmore: mmcgrath: just thought of something ill switch off plague on June 29 for FC-5
[15:46] dgilmore: skvidal: anyidea when you will get to rebuild the box?
[15:46] skvidal: dgilmore: not this week and probably not beginning of next since I'll be in orientation, etc
[15:46] mmcgrath: dgilmore: <nod>
[15:47] skvidal: but I'll be working again come next week
[15:47] skvidal: so it's a start
[15:47] skvidal: and I should be able to spend the time
[15:47] mmcgrath: cool
[15:48] mmcgrath has set the subject to Ibiblio Mirror - On hold
[15:48] mmcgrath: The ibiblio mirror is on hold for probably about a week while we hook don up with direct I2 access to our mirror in RDU.
[15:48] mdomsch: mmcgrath, pick set up the static route already
[15:48] mmcgrath: mdomsch: hmm, I'll have to check with don, he was under the impression he needed to wait a bit.
[15:49] mmcgrath: Ok, thats all I've got.
[15:49] mmcgrath has set the subject to Open Floor ----------------
[15:49] lmacken: word
[15:49] lmacken: I was wondering what you guys thought about having some sort of development environment for our webapps.
[15:49] lmacken: So, there are a handfull of people that are interested in hacking on bodhi, but due it's dependencies on koji and mash, it's extremely difficult to develop it anywhere other than PHX. I've currently been doing all of my development on publictest2, which has been working out great.
[15:49] lmacken: So a possibility for this is to have some Xen guest with a read-only mount of /mnt/koji and blocked out from the rest of PHX.
[15:49] mdomsch: lmacken, +1; /me misses publictest7
[15:50] lmacken: yeah, and honestly.. i have no idea how to start hacking on mirrormanager, smolt, etc
[15:50] lmacken: i think if we opened the doors a bit, our infrastructure could improve vastly
[15:50] lmacken: mdomsch: feel free to hack on publictest2 for now
[15:50] mmcgrath: lmacken: the main limiting facter on that is RAM, but we can set something up.
[15:51] lmacken: mmcgrath: cool
[15:51] mmcgrath: lmacken: we should probably setup more shared xen instances.
[15:51] abadger1999: lmacken: +1
[15:51] dgilmore: im going to start work on enabling secondary archs if anyone wants to help feel fee to talk to me
[15:52] dgilmore: mmcgrath: can we possibly get another vlan?
[15:52] jcollie: mmcgrath, could i get a xen guest for testing the git/vcs stuff?
[15:52] lmacken: mmcgrath: cool.. so what is the next action to getting this ready? creating a group for infrahackers and granting access on a restricted guest ?
[15:52] dgilmore: mmcgrath: so we can seperate the some guestd for this kind of thing
[15:52] mmcgrath: lmacken: well I'll need to find where we have RAM avaiable for the instances. Its item "Server Upgrades" on the wiki.
[15:52] mmcgrath: dgilmore: we should.
[15:53] wolfy has left ("When you are down and out something always turns up-and it is usually the noses of your friends." (n=lonewolf@fedora/wolfy))
[15:53] lmacken: mmcgrath: ok.. well, publictest2 has been my playground for the past few months.. any reason not to just start using that ?
[15:53] mmcgrath: jcollie: I think we can setup something, it'll be a bit
[15:53] mmcgrath: lmacken: not sure, I think it only has 512M ram right?
[15:53] lmacken: mmcgrath: i'm not sure
[15:54] dgilmore: mmcgrath: im pretty sure thats all it ahs
[15:54] dgilmore: has
[15:54] lmacken: mmcgrath: also, i noticed that you setup the security guest.. does bressers know about it yet ?
[15:54] mmcgrath: k, I'll try to find ways to consolidate some of our lesser machines into a bigger, sort of super machine.
[15:54] lmacken: mmcgrath: cool
[15:54] mmcgrath: lmacken: I think dgilmore did that
[15:55] lmacken: ah
[15:55] lmacken: dgilmore: is the security guest ready to go ?
[15:56] dgilmore: lmacken: not yet
[15:56] lmacken: dgilmore: k, just checking
[15:56] dgilmore: i need to add the security group to get shell access
[15:58] mmcgrath: solid
[15:58] mmcgrath: so anyone have anything else? If not I'll close the meeting in 30 seconds?
[15:58] mmcgrath: 10
[15:59] mmcgrath has set the subject to Meeting End -----------------------
i'm looking for some feedback on speeding up moinmoin after chatting
with ThomasWaldmann in #moin about speeding up page save operations:
(08:36:48 PM) idioteque123: i've come to this from the point of view
of ending up with a package for http://fedoraproject.org/wiki
(08:37:42 PM) ThomasWaldmann: idioteque123: how to proceed depends on
what you want and when
(08:38:31 PM) idioteque123: there aren't any well defined goals for
this in #fedora-admin just yet
(08:38:58 PM) ThomasWaldmann: if you want something soon, I would
suggest you work with 1.6. if you have more time, you could wait until
storage api stuff gets usable.
(08:39:40 PM) idioteque123: ok, well how about i start a page
somewhere in the moinmoin development wiki?
(08:39:44 PM) ThomasWaldmann: btw, the jabber branch introduces some
(08:40:10 PM) idioteque123: then get some feedback from the
fedora-admin people about 1.6 vs 1.7
(08:40:21 PM) ThomasWaldmann: idioteque123: yeah, good plan. or look
if there is already something, we discussed that somewhere before
(maybe on wiki, maybe on irc).
so, do people think i should work in the 1.6 code or the 1.7 code? i
note that there is already some work in the 1.7 branch for fedora
but i'm guessing we might want some results sooner than that?
also, i should create a page in the fedora wiki to track this?
i'll be around in for the next infrastructure meeting to get more feedback.
httpd on koji.fp.o got OOM killed a few minutes ago. I've stopped httpd to
recover memory and started it back up again. Things seem ok now, somebody
should investigate further.
Release Engineer: Fedora
It's the time to present myself. My name is Marek and I'm working for
one big software company right now. Before I was in HP support,
administrating HP-UX, linux and few AIXs. I really want help you to
maintenance fedora project infrastructures, basically from admin side,
but maybe form devel as well.
I think that it's maybe a good idea to build an escalation paths for
infrastructure, to assure that at any time, there will be someone to fix
things or get in touch with someone who can in case of problems.
Let me know how can I help and what do you think about my idea.
Marek Mahut Tel.: +420-532-294-111 (ex. 826-2046)
Fedora Ambassador GSM: +420-731-076-674
i have applied for a role in the infrastructure team. The
FAS has not been approved so far. is there anything else that needs to
be done... please let me know
Jose M Manimala
S6 Computer science and engineering
Rajagiri School Of Engineering And Technology
So I changed cacti to use cacti internal auth (instead of http auth).
So its public now:
The bad news is I have to create users individually now in cacti. If
you need access to alter things just let me know and I'll get you set up.
-----BEGIN PGP SIGNED MESSAGE-----
I was looking performing some security patch update via RHN, anyways I
noticed that four of our servers don't have any iptables implemented
I am not sure why lockbox doesn't have iptables implemented as it is the
machine that contains all security logs and should be one of our most
I reckon this should be an issue discussed at our next meeting. I also
looked at the proxy[1,2] servers and the iptables implemented (could be
tided up) also app[1,2,3] had some basic that could be re-written.
anyways I thought this should be brought to everyones attention.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
-----END PGP SIGNATURE-----
Is anyone looking into writing a web interface that can do custom spins
using pungi and livecd-tools in the background? Revisor is useful as a
graphical wrapper but it requires Fedora to be installed already.
Having a web interface would allow anyone to select the package groups
and individual packages and get a ISO image for download. I posted to
fedora-infrastructure list before but we need a web app before talking
about deploying it.
There is new effort launched in http://respins.org to provide a forum
and encourage community respins or derivatives of Fedora and they have
expressed a interest in this too.