Hi,
It's me again, about Retro Changelog trimming :-(. Last time it was
about the maxage
configuration, for which I created an issue [1].
This time, the problem is that of a deadlock. When I have maxage set
to 2d (the
default), then soon after restart the server starts to do the
trimming.
Unfortunately it quickly runs into a deadlock. All accesses to the
server (e.g ldapsearch)
hang forever. And because this is a replica, the other servers are
complaining too.
Looking at a gdb stack trace I see the following.
$ sudo cat
gdb-trace-ns-slapd-4.txt | grep -E '^(Thread|#[01] .*lock)'
Thread 41 (Thread 0x7fefa3e72700 (LWP
170190)):
#0 0x00007fef9f9b52f5 in
pthread_rwlock_wrlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2750 in map_wrlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 40 (Thread 0x7feef147d700 (LWP
170184)):
Thread 39 (Thread 0x7feeef2f9700 (LWP
170178)):
Thread 38 (Thread 0x7feef1c7e700 (LWP
170171)):
Thread 37 (Thread 0x7feef247f700 (LWP
170169)):
Thread 36 (Thread 0x7feef37ff700 (LWP
170166)):
Thread 35 (Thread 0x7feef67ff700 (LWP
170165)):
Thread 34 (Thread 0x7feef75fe700 (LWP
170164)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 33 (Thread 0x7feef7dff700 (LWP
170163)):
Thread 32 (Thread 0x7feef89fe700 (LWP
170162)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 31 (Thread 0x7feef91ff700 (LWP
170161)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 30 (Thread 0x7feef9dfe700 (LWP
170160)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 29 (Thread 0x7feefa7ff700 (LWP
170159)):
Thread 28 (Thread 0x7feefb7ff700 (LWP
170158)):
Thread 27 (Thread 0x7feefc3fe700 (LWP
170157)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 26 (Thread 0x7feefcdff700 (LWP
170156)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 25 (Thread 0x7feefe1fe700 (LWP
170155)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 24 (Thread 0x7feefebff700 (LWP
170154)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 23 (Thread 0x7feeff7da700 (LWP
170153)):
Thread 22 (Thread 0x7feefffdb700 (LWP
170152)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 21 (Thread 0x7fef007dc700 (LWP
170151)):
Thread 20 (Thread 0x7fef00fdd700 (LWP
170150)):
#0 0x00007fef9f9b4ec2 in
pthread_rwlock_rdlock () at target:/lib64/libpthread.so.0
#1 0x00007fef8e9f2612 in map_rdlock () at
target:/usr/lib64/dirsrv/plugins/schemacompat-plugin.so
Thread 19 (Thread 0x7fef02fd9700 (LWP
170148)):
Thread 18 (Thread 0x7fef037da700 (LWP
170147)):
Thread 17 (Thread 0x7fef03fdb700 (LWP
170146)):
Thread 16 (Thread 0x7fef049e3700 (LWP
170145)):
Thread 15 (Thread 0x7fef051e4700 (LWP
170144)):
Thread 14 (Thread 0x7fef059e5700 (LWP
170143)):
Thread 13 (Thread 0x7fef071ff700 (LWP
170140)):
Thread 12 (Thread 0x7fef081ff700 (LWP
170139)):
Thread 11 (Thread 0x7fef08ffe700 (LWP
170138)):
Thread 10 (Thread 0x7fef097ff700 (LWP
170137)):
Thread 9 (Thread 0x7fefa3e93700 (LWP
170136)):
Thread 8 (Thread 0x7fef0a5ff700 (LWP
170135)):
Thread 7 (Thread 0x7fef0b3ff700 (LWP
170134)):
Thread 6 (Thread 0x7fef0ca09700 (LWP
170133)):
Thread 5 (Thread 0x7fef0d20a700 (LWP
170132)):
Thread 4 (Thread 0x7fef0da0b700 (LWP
170131)):
Thread 3 (Thread 0x7fef0e20c700 (LWP
170130)):
Thread 2 (Thread 0x7fef0ea0d700 (LWP
170129)):
Thread 1 (Thread 0x7fefa3f98240 (LWP
170127)):
The version info:
389-ds-base-libs-1.4.3.28-6.module_el8.6.0+1102+fe5d910f.x86_64
389-ds-base-1.4.3.28-6.module_el8.6.0+1102+fe5d910f.x86_64
For the time being I have changed maxage to 200d, to avoid trimming,
to avoid deadlock.
But in the long run it causes to changelog to grow and grow. One
server has over 2GB,
another server has already more than 4GB in the changelog db.
[1] https://github.com/389ds/389-ds-base/issues/5368
--
Kees