Hello all,
Need some help with tuning and crash debugging. We’re running Fedora-Directory/1.0.4 B2006.312.1539. The problem is on our “Dedicated Consumer” machine running on RHEL 5. We have over ~150,000 users authenticating against our FDS systems. System resources are not a problem (~.39 load, free memory, 92k swap)
For months, the system is solid without any issues then we seem to get a large spike in traffic and FDS crashes. I run Monit so the service is restarted automatically but I cannot figure out why the service keeps crashing.
FDS was setup and tuned based off: http://directory.fedoraproject.org/wiki/Performance_Tuning#Linux
I have reviewed http://directory.fedoraproject.org/wiki/FAQ#Debugging_Crashes as well, but some of that is over my head. I have turned buffering off and increased the logging level in the LDAP config.
Here is our “monitor” script output:
version: 1
dn: cn=monitor
objectClass: top
objectClass: extensibleObject
cn: monitor
version: Fedora-Directory/1.0.4 B2006.312.1539
threads: 30
currentconnections: 19
totalconnections: 11918
dtablesize: 8192
readwaiters: 0
opsinitiated: 43703
opscompleted: 43702
entriessent: 16086
bytessent: 2911011
currenttime: 20110805164243Z
starttime: 20110805114053Z
nbackends: 2
Here is our “Access Log Analyzer” summary for a 24 hour period:
---------------------------------------------------------------
Access Log Analyzer 6.0
Filename Total Lines Lines processed
---------------------------------------------------------------
/opt/fedora-ds/slapd/logs/access 298225 298231
----------- Access Log Output ------------
Restarts: 6
Total Connections: 39720
Peak Concurrent Connections: 84
Total Operations: 95471
Total Results: 95393
Overall Performance: 99.9%
Searches: 48215
Modifications: 167
Adds: 551
Deletes: 2
Mod RDNs: 0
6.x Stats
Persistent Searches: 0
Internal Operations: 0
Entry Operations: 0
Extended Operations: 845
Abandoned Requests: 0
Smart Referrals Received: 0
VLV Operations: 0
VLV Unindexed Searches: 0
SORT Operations: 0
SSL Connections: 0
Entire Search Base Queries: 0
Unindexed Searches: 6
FDs Taken: 39720
FDs Returned: 39657
Highest FD Taken: 93
Broken Pipes: 0
Connections Reset By Peer: 0
Resource Unavailable: 10872
- 10872 (T1) Idle Timeout Exceeded
Binds: 45691
Unbinds: 27987
LDAP v2 Binds: 15694
LDAP v3 Binds: 29997
SSL Client Binds: 0
Failed SSL Client Binds: 0
SASL Binds: 0
Directory Manager Binds: 0
Anonymous Binds: 16346
Other Binds: 29345
---------------------------------------------------------------
In FDS console:
-- Configuration > Performance tab: Size Limit: 2000, Time Limit: 3600, Idle Timeout: 60, Max file descriptors: 8192.
-- Configuration > Data > Database Link Settings > Connection Management: Max TCP Connections: 10, Bind timeout: 20, Max binds per connection: 20, Timeout before abandon: 10, Max LDAP Connections: 20, Max bind retries: 3, Max operations per connection: 5, connection life: 60.
We have talked about moving to the latest 389 Directory packages and I have a migration process tested out so it’s a matter of getting the OK and time but I doubt the upgrade will solve our crashing problem. It seems to me we are hitting some limits that just haven’t been accounted for yet and that is where I need help.
Any suggestions on how to proceed with stopping these crashes is welcomed! Thanks for reading.
Trevor