-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Add HW.
You need a dedicated load balanced env if you want to ever run the
primary bits. We have 5 Dell servers for PRX, 4 APP that fell over. On
top of that the Load Balancers died one time due to load - and we were
not even serving pages, just redirects!
These are Cisco CSM blades that failed today. Not small Load Balancers.
Probably need to define well, the roles people will play for
administration, so that you do not have a free for all. Maybe even
shifts of people so Seth can take a shower or get food :)
seth vidal wrote:
Hey guys,
So some of you have heard that the load from today was pretty close to
crippling for a while. We got it under 'control' for a bit but it wasn't
happy. Mike suggested making some of our pages in the wiki static so we
pulled them out using firefox and then put them onto the site using
apache redirects. That worked out okay to bring the load down. We're
still get beaten pretty badly but at least the load is manageable.
Let's assume for now that the next N releases will be just like this.
That the world will melt down. I think we should probably put together
some docs on what steps to take. Here are some items we did today:
1. iptables rate limiting on fpserv: 2-4 new connection per ip for every
10-20 seconds.
2. look at the logs, figure out where all the hits are going and make
those pages static. You can do this by using firefox to save the page
(it grabs images and css, too)
3. turn off any unnecessary bits: zope (if its not being used, rsyncd)
4. trim out your log outputs. Decrease the number of things writing out
to disk anywhere
5. get comfortable with /etc/init.d/httpd restart - just to get the load
back under control
What else should we be adding?
-sv
_______________________________________________
Fedora-infrastructure-list mailing list
Fedora-infrastructure-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
- --
========================================================
= Stacy J. Brandenburg Red Hat Inc. =
= Manager, Network Operations sbranden(a)redhat.com =
= 919-754-4313
http://www.redhat.com =
========================================================
Fingerprint
03F7 43BE 1150 CCFA F57B 54DD AEDB 1C27 1828 D94D
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with Fedora -
http://enigmail.mozdev.org
iD8DBQFFPoNnrtscJxgo2U0RAp6rAKDGX11u0J2onLKYfyXlsC1MUhjC/wCg1Tqp
Y/E2QQi7Lwv8NKm2lOktLk4=
=6+yT
-----END PGP SIGNATURE-----