Meeting Re: Eucalyptus hosting
by Garrett Holmstrom
Hi,
At the infrastructure meeting yesterday Seth brought up the topic of
cloud hosting for Fedora services. With EC2 and Rackspace being
closed platforms, I suggested Eucalyptus as an alternative. I wasn't
able to find any hosting providers in the US that use it, but Rich
Wolski, the CTO of Eucalyptus Systems, expressed an interest in
chatting with you infrastructure folks about whether or not we at
Eucalyptus can just host it ourselves, depending on Fedora's needs and
such.
Would there be a good time and place to discuss this? The weekly IRC
meeting? (For Rich's benefit: the weekly IRC meeting is at noon PST
on Thursdays in #fedora-meeting on irc.freenode.net.) Another IRC
meeting in case the weekly one would be too pressed for time? A
conference call? Might an email thread be better?
--
Garrett Holmstrom
13 years, 2 months
Wiki extension request
by Adam Williamson
Hi, everyone. I'd like to ask about adding an extension to the Fedora
wiki instance.
Many of you may have noticed the package-specific test case project I've
been working on lately - https://fedorahosted.org/fedora-qa/ticket/154 .
Part of the plan for this project is to allow tools such as Bodhi to use
mediawiki categories to figure out certain sets of test cases. One
important one is 'all critical path test cases for package foo'.
However, this requires the ability to do multi-category search in the
wiki - find all pages that are members of two (or more) different
categories.
We've identified a couple of extensions which can potentially achieve
this:
http://www.mediawiki.org/wiki/Extension:Multi-Category_Search
http://www.mediawiki.org/wiki/Extension:DynamicPageList_%28Wikimedia%29
I'd like to ask that we look at including one of these (or someone tell
me that we already do :>).
There are a couple of concerns that I don't have sufficient expertise to
address. There would need to be some kind of, I guess, API access to the
searches, so that an external tool could do the search - both extensions
seem rather geared to doing the search entirely within the Wiki, for
instance, including the results of the search in another page. I don't
know how to tell which, if either, extension is most suited to this. The
other concern is recursion - does either extension handle this case:
page is a member of categories A and Z, but not directly of category B
category A is a member of category B
if searching for 'all pages that are in category B and Z', does the page
show up? For our purposes I'd like that the be the case.
If anyone can help with this it'd be greatly appreciated! Thanks.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org
http://www.happyassassin.net
13 years, 2 months
Querying FAS details through IRC
by Misha Shnurapet
Hello.
It is possible to query contributors' information on IRC with a command that fetches data from FAS. While it's OK to provide extensive information on users through the web interface (which is now only possible for the community members, AFAIK, as it requires to log in), I think it might come reasonable not to provide email addresses on such queries on IRC.
--
Best regards,
Misha Shnurapet, Fedora Project Contributor
http://fedoraproject.org/wiki/User:Shnurapet
shnurapet AT fedoraproject.org, GPG: 00217306
13 years, 2 months
Outage notes
by Seth Vidal
Hi Everyone,
I took some notes while we were rebooting boxes I wanted to share them
with everyone for future outages.
Ordering of the bounces:
1. xen14: puppet is on there and if that is back up first we have a
place to stand for pushing out any changes (dns changes for example via
puppet) - xen14 takes about 4 minutes to restart/POST
2. xen15: bastion01, db02 are on there - same 4 minute restart window
once this is up you'll want to logout of bastion02 and into bastion01
so you have a firm place to do the xen05 reboots from which will take
out bastion02.
3. edit dns on puppet to remove proxy01 from the wildcard/roundrobin and
push that to the ns* servers and verify.
4. xen05: bastion02 (openvpn), proxy01 - 4-5 minutes for this machine to
restart.
once xen05 is completely up log back in and verify the vpn is back
online
5. edit dns remove all the other proxy hosts and put proxy01 back in.
Push and verify
6. virthost01 - I had to halt each of the kvms from a login - virsh
shutdown didn't work. - 4 minute restart time on the hw.
Note: make sure virthost01 is completely up - especially fas03. since
taking down virthost02 next will take out fas02- you want to make sure
you don't leave fas01 all by itself.
7. virthost02 - fas02 was not setup to autostart - that's now fixed.
8. virthost13 - uneventful
9. xen03 - spin01 spewed lots of umount issues - those are from the spin
creation paths - they can be safely ignored
- fas01.stg was running on xen03 according to the logs but
there's no definition for it on the system - so not sure what the story
is there.
- neither of the other staging hosts were set to autostart
10. xen04: we apparently have a number of hosts w/only one dns record
internally and they point to ns03 only. B/c when ns03 went away - lots
of things got VERY VERY SLOW trying to resolve names. This is on my list
to address. You must wait for xen04 to be completely up and ns03 running
before you can take down xen07. Otherwise we'll be w/o dns internally to
phx.
11. xen07: iscsi disks didn't come up right away - this kept ns04 from
coming up immediately - needed to run /etc/init.d/iscsi start and they
showed up.
12. xen09: uneventful
13. xen10: log01 needed an fsck b/c of the time since last mount - this
took a long time.
14. xen11: secondary1 needed an fsck. also a 5-6 minute hw reboot time.
15. xen12: db1->db01 naming change kept it from coming up at boot b/c of
the 'auto' symlink to db1. db01 had to fsck
16. cnode01 - 6-7 minute reboot time - nothing was set to autostart in
xen - this is now fixed - autoqa01 and dhcp02 are set to autostart
17. db03: fsck took FOREVER to complete and this takes a lot of things
done - for the future move db03 reboot higher up the stack, just in
case. This machine's restart/POST time is REALLY high like 7-10 minutes.
The console for it is less than forthcoming, too.
18. backup01: uneventful
At this point internal was back online - except for the build xen
systems and servers.
External hosts:
19. - bodhost01: 5-6 minute machine reboot time
- people01 - uneventful.
- ibiblio01 - 5-7 minute machine reboot time. uneventful
20. - internetx01: uneventful
- osuosl01: uneventful
21. - sb2 - must wait for ibiblio01 to be up b/c of not having any
external name servers
- sb3 - uneventful
- sb4 - hosted1 listed more 'maxmem' in its config that sb4 had
available - so that had to be edited down. Not sure how that EVER
started
- sb5 - uneventful
22. telia01 - proxy5 did not restart on its own - unknown as to WHY yet
- but it did start manually.
- retrace01 was not set to autostart
tummy1 - uneventful
Now all the proxy* rebooting is over so we can:
23. edit dns: put the other proxy hosts in the wildcard/RR - push and
verify
Build boxes:
- bxen03 had koji2 listed in its set of hosts - but it wasn't running.
This led to some confusion as to how to start the hosts on bxen03 b/c
of insufficient memory for all guests. Eventually I realized bxen04 is
where koji02 was running and that the left over guest file was never
cleaned up on bxen03.
Things to think about post-outage:
- check all the raid arrays for lost disks - we saw this a couple of
times - it's not pleasant.
- check for downed vpns and/or broken resolution - we need to get a
firm handle on why this is a hassle so often.
Overall things to think about for the future:
1. dumping a complete virsh list - including how much memory is actually
being used per vm per server before we start reboots
2. checking what disks need fscks because of mounted time and doing
those earlier or separately.
3. verifying that all running vms are:
a. intended to be running
b. have a config file
c. are set to autostart
4. verifying that all NOT running vms are:
a. intended to be off
b. are NOT set to autostart
thoughts welcome.
-sv
13 years, 2 months
Turbogears downgraded on fas servers
by Toshio Kuratomi
I've re-downgraded TurboGears to 1.0.9 on the fas servers. Another
incompatibility was discovered i nthe TG-.1.1.x series. EPEL6 is going out
of beta soon so on Monday I'll need to decide whether to revert TG in EPEL6
to TG-1.0.x or work on fixing fas further.
-Toshio
13 years, 2 months
REMINDER: publictest box motd [this means you :)]
by Stephen John Smoogen
Ok when testing a project it is important to update the /etc/motd file
with the following information:
[fas username] - [date started] [date updated]
project title - project summary
directories/areas looked at.
Example:
Project Owner: smooge - Started: 2010-12-31 Updated: 2010-01-30
gnusong - a project to digitize the reading of all source code to a dance beat.
/usr/local/bin, /srv/web/songs
1) Systems that DO NOT have updated motd's will be wiped, reinstalled
to a fresh state and turned off until an RFR is needed for it. Systems
whose last update was over 90 days will also be subject to
wiping/reinstall/turn off.
2) publictest systems are NOT backed up. they are test/scratch systems
that are meant for eventual integration with some sort of
development/staging infrastructure.
3) due to communal nature of systems, people need to work together on
them to make sure they are not overwriting and beating each other up.
--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren
13 years, 2 months
delta rpm compose debugging change
by Kevin Fenzi
Greetings.
Of late openoffice.org and a few other packages have been not
generating delta rpms in pushes. ;(
They error with:
Error genDeltaRPM for chunkd: exitcode was 256 - Reported Error: bad cpio archive
But looking later at both rpms, things seem fine. ;(
I've talked with jdieter and we would like to add in more debugging:
<jdieter> nirik: We're still having the "bad cpio archive" with openoffice.org updates
<jdieter> I have a patched version of deltarpm available at: http://www.lesloueizeh.com/jdieter/deltarpm-3.6-0.1.20101230git.fc14.src.rpm
<jdieter> If run with -vv, it will report old rpm, new rpm, and tell us what the actual first six bytes of what's actually in the rpm
<jdieter> vs the 070701 that it's supposed to be
<jdieter> I don't have an el6 system, so it will have to be rebuilt for releng2
<jdieter> /usr/lib/python2.4/site-packages/createrepo/deltarpms.py needs to modified as well
<jdieter> The line with "makedeltarpm -v" needs to be changed to "makedeltarpm -vv"
epel5 scratch build at:
http://koji.fedoraproject.org/koji/taskinfo?taskID=2694315
(releng02 is rhel5).
If I can get enough +'s I'd be happy to put this in place whenever.
Otherwise we can wait until next week after the freeze.
Thoughts? Comments? Other ideas?
kevin
13 years, 2 months
Back
by Stephen John Smoogen
I am back and starting to catch up with emails. I ended up with almost
0 internet access last week so thanks to those who covered and watched
ibiblio go up and down ( I was drawing talking with Crush the Turtle
at the time :)).
--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren
13 years, 2 months