On 10/13/2011 12:30 PM, Justin Gronfur wrote:
Well, I figured it out!
After removing the Java stack completely from the equation, I noticed
that I was getting worse, but more consistent performance. The worse
performance indicated something to do with caching, so after I dug
into the Java code for a while I found that we were caching certain
object classes (therefore making them not need to be rebuilt from
ldap). The timeout on that cache was 5 minutes.
So during load testing, the tests run great until everything in that
cache expires at once making all 15 threads hit 389 to rebuild the
cache entries at the exact same time. My data sample used for testing
involved about 150 objects being cached, which takes 150 queries to
generate. So when the cache expires, 15*150=2250 queries hit 389 at
the exact same time which causes it to seemingly lock up for 20
seconds while it handles all of that.
I am still surprised that 389 can't handle that better, but I haven't
spent much time tweaking settings or indexes for performance. I'm
sure that we are running very sub-optimally right now.
The stack traces you
provided give us some clues. I would suggest
lowering nsslapd-threadnumber in cn=config to see if that makes a
difference - one factor could be excessive thread contention based on
your stack traces.
You can use the logconv.pl script to analyze your access log to see if
you need to tune your indexes or other performance related settings such
as nsslapd-lookthroughlimit or nsslapd-idlistscanlimit.
Thanks,
Justin