What is the test filter like?
Can we see a sanitized sample of the access log with the SRCH and RESULT?

If using SSL, review the output of
cat /proc/sys/kernel/random/entropy_avail

Do we have replication? (and large attribute values?)

You may want to run the "dbmon.sh" script to monitor cache usage for db cache and entry cache, try to capture a few samples of line about  dbcachefree and userroot:ent (if the db with the problems is userroot), when the searches are becoming too long, like this example:
INCR=1 HOST=m2.example.com BINDDN="cn=directory manager" BINDPW="password" VERBOSE=2 /usr/sbin/dbmon.sh

and review the ns-slapd errors and system messages log files for any unusual activity.

what is the ns-slapd memory foot print from restart to slow responses?
any "too high" disk i/o? (or "bad" ssd?)

Thanks,
M.

On Tue, Nov 15, 2016 at 11:40 AM, Gordon Messmer <gordon.messmer@gmail.com> wrote:
I'm trying to track down a problem we are seeing on two relatively lightly used instances on CentOS 7 (and previously on CentOS 6, which is no longer in use).  Our servers have 3624 entries according to last night's export (we export userRoot daily).  There are currently just over 400 connections established to each server.

We have a local cron job that runs every 5 minutes that performs a simple query.  If it takes more than 7 seconds to get an answer, the attempt is aborted and another query issued.  If three consecutive test fail, the directory server is restarted.

The issue we're seeing is that the longer the system is up, the more often checks will fail.  Restarting the directory does not resolve the problem.  Our servers have currently been up for 108 days, and are restarting the service several times a day, as a result of the checks.  Only if we reboot the systems does the problem subside.

CPU utilization seems relatively high for such a small directory, but it's not constant.  I tried to manually capture a bit of data with strace during a period when CPU use was bursting high.  During a capture of maybe two seconds, I saw most of the CPU time was spent in futex.  usecs/call was fairly high for calls to futex and select, as detailed below.

Since restarting the service doesn't fix the problem, it seems most likely that this is an OS bug, but I'm hoping that the list can help me identify other useful data to track down the problem.  Does anyone have any suggestions for what I can capture now, while I can sometimes observe the problem?  If I reboot, it'll take months before I can get any new data.


% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 74.61    4.505251        3590      1255       340 futex
 17.65    1.065548        6660       160           select
  4.41    0.266344       88781         3         2 restart_syscall
  3.07    0.185566          50      3718           poll
  0.10    0.006185           2      3610           sendto
  0.09    0.005189        5189         1           fsync
  0.04    0.002134          37        58           write
  0.03    0.001618          27        61           setsockopt
  0.00    0.000111           3        36           recvfrom
  0.00    0.000078           1        57           read
  0.00    0.000014          14         1           fstat
  0.00    0.000003           2         2           accept
  0.00    0.000003           1         6           fcntl
  0.00    0.000002           1         2           getsockname
  0.00    0.000001           1         2           close
------ ----------- ----------- --------- --------- ----------------
100.00    6.038047                  8972       342 total
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org