This outage is considered finished and closed at the moment.
Due to some change in network, DNS traffic was not being properly
moving between Fedora network and Red Hat filer. It appears that in
doing so the filer's NFS permission caches were being flooded with bad
lookups thus increasing CPU and lowering throughput.
On Wed, Dec 15, 2010 at 15:42, Stephen John Smoogen <smooge(a)gmail.com> wrote:
Due to problems with some NFS in our PHX2 facility we are
experiencing
diminished capacity to several of our services. We are working with
our provider and engineers on how to deal with this issue soon.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:
date -d '2010-12-15 22:00 UTC'
Reason for outage:
NFS operations with filer are peaking below expected rates causing
hangs on NFS clients.
Affected Services:
BFO -
http://boot.fedoraproject.org/
Buildsystem -
http://koji.fedoraproject.org/
CVS / Source Control
Main Website -
http://fedoraproject.org/
Mirror List -
https://mirrors.fedoraproject.org/
Mirror Manager -
https://admin.fedoraproject.org/mirrormanager/
Package Database -
https://admin.fedoraproject.org/pkgdb/
Unaffected Services:
Bodhi -
https://admin.fedoraproject.org/updates/
DNS -
ns1.fedoraproject.org,
ns2.fedoraproject.org
Docs -
http://docs.fedoraproject.org/
Email system
Fedora Account System -
https://admin.fedoraproject.org/accounts/
Fedora Community -
https://admin.fedoraproject.org/community/
Fedora Hosted -
https://fedorahosted.org/
Fedora People -
http://fedorapeople.org/
Fedora Talk -
http://talk.fedoraproject.org/
Smolt -
http://smolts.org/
Spins -
http://spins.fedoraproject.org/
Start -
http://start.fedoraproject.org/
Torrent -
http://torrent.fedoraproject.org/
Translation Services -
http://translate.fedoraproject.org/
Wiki -
http://fedoraproject.org/wiki/
Ticket Link:
https://fedorahosted.org/fedora-infrastructure/ticket/2516
Contact Information:
Please join #fedora-admin in
irc.freenode.net or respond to this email to
track the status of this outage.
--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren
--
Stephen J Smoogen.
"The core skill of innovators is error recovery, not failure avoidance."
Randy Nelson, President of Pixar University.
"Let us be kind, one to another, for most of us are fighting a hard
battle." -- Ian MacLaren