I recently reinstalled a couple of our freeipa replicas and they're both falling over
with the same error. They run for a few minutes - as little as one, or up to an hour, and
then fall over with thousands of errors like this:
ERR - accept_and_configure - PR_Accept() failed, Netscape Portable
Runtime error -5971 (Process open FD table is full.)
Eventually we get some db errors and dirsrv crashes (which presumably is related to the FD
table being full). I'm unable to determine why the table is filling.
While they're up, the 2 failing replicas work fine (web, ldap) and seem to have all
the appropriate info. The server they're being replicated from is fine, no notable
errors, but it only has about 180 servers hitting it. There's another replica with no
clients that works fine, no issues. These two servers used to work fine before I
reinstalled the replicas (I have not changed the OS). It's a pretty small directory
and we've got a couple thousand servers hitting the two failing replicas. They were
recently updated from ipa-server 4.4 to 4.6, and used to work fine, the errors started
after I rebuilt them. (BTW, I'm not the person who built the originals.)
The logs are not very giving me much in the way of clues. File handles are set in
/etc/sysconfig/dirsrv to 32k, which should be more than enough, it was previously set to
16k. It hasn't gotten any busier than it used to be, so I don't feel like the file
handle issue is simply due to being sized wrong. Debug logging generated a lot of output
but nothing I could clearly identify as being an issue.
I'm at a loss. What should I be checking?
CentOS Linux release 7.6.1810 (Core)