Hi,
The deadlock should happen quite rarely but remain a possibility.
Shouldn't we address this deadlock scenario in others ipa scripts that
are doing DS admin tasks (import, index). For example disabling/enabling
during those tasks ?
regards
thierry
On 05/03/2018 09:16 PM, Rob Crittenden via FreeIPA-users wrote:
> thierry bordaz wrote:
>> Hi Soler,
>>
>> Thanks for the information.
>> So indexing is hanging because SC cache_init is running, the SC
>> cache_init is hanging because SSSD is not started, SSSD is not
>> started possibly because indexing prevents to get read access to the
>> backend ("Backend is offline" TBC).
>>
>> An option would be to disable SC during upgrade phase ? Rob ?
>
> Sure, disabling SC would be a way to get things running again.
>
> To do this:
>
> 1. Be sure dirsrv is not running
> 2. edit /etc/dirsrv/slapd-YOUR-REALM/dse.ldif
> 3. find the entry for cn=Schema Compatibility,cn=plugins,cn=config
> 4. change nsslapd-pluginEnabled from on to off
> 5. try the upgrade again
>
> Assuming the upgrade is successful you can re-enable it post-install
> with:
>
> # ipa-compat-manage enable
>
> rob
>
>>
>> regards
>> thierry
>>
>> On 05/03/2018 11:35 AM, SOLER SANGUESA Miguel wrote:
>>> Hello Thierry,
>>>
>>> The version is: slapi-nis-0.56.0-8.el7.x86_64
>>>
>>> And the errors are:
>>> [02/May/2018:13:04:30.089731032 +0200] - ERR - schema-compat-plugin
>>> - group "XXXXXX(a)ipa.example.org" does not exist because SSSD is
>>> offline.
>>> [02/May/2018:13:04:30.093169411 +0200] - ERR - schema-compat-plugin
>>> - waiting for SSSD to become online...
>>>
>>> SSSD service is up, but as the ipa services are not (all) running,
>>> the ipa domain is down, logs when started sssd shows:
>>> sssd[be[ipa.example.org]][823]: Backend is offline
>>>
>>> and logs sssd_nss.log & sssd_sudo.log have hundreds of:
>>> [sss_dp_get_reply] (0x0010): The Data Provider returned an error
>>> [org.freedesktop.sssd.Error.DataProvider.Offline]
>>>
>>> Thanks & Regards.
>>>
>>> -----Original Message-----
>>> From: thierry bordaz <tbordaz(a)redhat.com>
>>> Sent: Thursday, May 03, 2018 11:25
>>> To: SOLER SANGUESA Miguel <solerm(a)unicc.org>; Rob Crittenden
>>> <rcritten(a)redhat.com>; FreeIPA users list
>>> <freeipa-users(a)lists.fedorahosted.org>
>>> Cc: Ludwig Krispenz <lkrispen(a)redhat.com>
>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
>>> from 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>>>
>>> me again ...
>>>
>>> If it exists some logs (/var/log/dirsrv/slapd-<xx>/errors) during
>>> the hanging period it could also indicate the reason what Schema
>>> compat was waiting for (SSSD ?)
>>>
>>> On 05/03/2018 10:38 AM, SOLER SANGUESA Miguel wrote:
>>>> hello,
>>>>
>>>> Yesterday my ssh console closed the connection, so I had to start
>>>> again the "ipa-server-upgrade", but the result is more or less
the
>>>> same:
>>>> # ipa-server-upgrade
>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>>>> [1/10]: stopping directory server
>>>> [2/10]: saving configuration
>>>> [3/10]: disabling listeners
>>>> [4/10]: enabling DS global lock
>>>> [5/10]: starting directory server
>>>> [6/10]: updating schema
>>>> [7/10]: upgrading server
>>>>
>>>> But now, the lines that are repeated on the access log are:
>>>> [03/May/2018:10:33:27.969889221 +0200] conn=6 op=79094 SRCH
>>>>
base="cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config"
>>>> scope=0 filter="(objectClass=*)" attrs="nstaskstatus
nstaskexitcode"
>>>> [03/May/2018:10:33:27.970146545 +0200] conn=6 op=79094 RESULT err=0
>>>> tag=101 nentries=1 etime=1.0000687740
>>>>
>>>> And it is on the same state than the other:
>>>> # ldapsearch -H ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket -b
>>>> cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config -s
>>>> base SASL/EXTERNAL authentication started SASL username:
>>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>>>> SASL SSF: 0
>>>> # extended LDIF
>>>> #
>>>> # LDAPv3
>>>> # base
>>>>
<cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config>
>>>> with scope baseObject # filter: (objectclass=*) # requesting: ALL #
>>>>
>>>> # indextask_l_137445500911864330_4055, index, tasks, config
>>>> dn: cn=indextask_l_137445500911864330_4055,cn=index,cn=tasks,cn=config
>>>> objectClass: top
>>>> objectClass: extensibleObject
>>>> nsindexattribute: l
>>>> nsinstance: userRoot
>>>> cn: indextask_l_137445500911864330_4055
>>>> nstaskcurrentitem: 0
>>>> nstasktotalitems: 1
>>>>
>>>> # search result
>>>> search: 3
>>>> result: 0 Success
>>>>
>>>> # numResponses: 2
>>>> # numEntries: 1
>>>>
>>>> The pstack of the proces: /usr/bin/python2
>>>> /usr/sbin/ipa-server-upgrade # pstack 9805
>>>> #0 0x00007fb906629be3 in __select_nocancel () from /lib64/libc.so.6
>>>> #1 0x00007fb8ff5e0000 in time_sleep () from
>>>> /usr/lib64/python2.7/lib-dynload/timemodule.so
>>>> #2 0x00007fb90730acf0 in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #3 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #4 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #5 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #6 0x00007fb90730d03d in PyEval_EvalCodeEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #7 0x00007fb90730a53c in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #8 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #9 0x00007fb90730d03d in PyEval_EvalCodeEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #10 0x00007fb90730a53c in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #11 0x00007fb90730d03d in PyEval_EvalCodeEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #12 0x00007fb90730a53c in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #13 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #14 0x00007fb90730d03d in PyEval_EvalCodeEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #15 0x00007fb90730a53c in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #16 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #17 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #18 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #19 0x00007fb90730a6bd in PyEval_EvalFrameEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #20 0x00007fb90730d03d in PyEval_EvalCodeEx () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #21 0x00007fb90730d142 in PyEval_EvalCode () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #22 0x00007fb90732657f in run_mod () from /lib64/libpython2.7.so.1.0
>>>> #23 0x00007fb90732773e in PyRun_FileExFlags () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #24 0x00007fb9073289c9 in PyRun_SimpleFileExFlags () from
>>>> /lib64/libpython2.7.so.1.0
>>>> #25 0x00007fb907339b7f in Py_Main () from /lib64/libpython2.7.so.1.0
>>>> #26 0x00007fb9065563d5 in __libc_start_main () from /lib64/libc.so.6
>>>> #27 0x000000000040066e in _start ()
>>>>
>>>>
>>>> And the pstack from the process: /usr/sbin/ns-slapd -D
>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG -i
>>>> /var/run/dirsrv/slapd-IPA-EXAMPLE-ORG.pid
>>>> # pstack 15536
>>>> Thread 50 (Thread 0x7f5bc0ae9700 (LWP 15539)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 49
>>>> (Thread 0x7f5bc02e8700 (LWP 15540)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 48
>>>> (Thread 0x7f5bbfae7700 (LWP 15541)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 47
>>>> (Thread 0x7f5bbf2e6700 (LWP 15542)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 46
>>>> (Thread 0x7f5bbeae5700 (LWP 15543)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 45
>>>> (Thread 0x7f5bbe2e4700 (LWP 15544)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 44
>>>> (Thread 0x7f5bbdae3700 (LWP 15545)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 43
>>>> (Thread 0x7f5bbd2e2700 (LWP 15546)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 42
>>>> (Thread 0x7f5bbcae1700 (LWP 15547)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 41
>>>> (Thread 0x7f5bbc2e0700 (LWP 15548)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 40
>>>> (Thread 0x7f5bbbadf700 (LWP 15549)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 39
>>>> (Thread 0x7f5bbb2de700 (LWP 15550)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 38
>>>> (Thread 0x7f5bbaadd700 (LWP 15551)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 37
>>>> (Thread 0x7f5bba2dc700 (LWP 15552)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 36
>>>> (Thread 0x7f5bb9adb700 (LWP 15553)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 35
>>>> (Thread 0x7f5bb92da700 (LWP 15554)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d33e63 in worker_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 34
>>>> (Thread 0x7f5bb8ad9700 (LWP 15555)):
>>>> #0 0x00007f5bd18a6113 in epoll_wait () at /lib64/libc.so.6
>>>> #1 0x00007f5bd1d9c803 in epoll_dispatch () at
>>>> /lib64/libevent-2.0.so.5
>>>> #2 0x00007f5bd1d883ea in event_base_loop () at
>>>> /lib64/libevent-2.0.so.5
>>>> #3 0x00007f5bd4d34eae in ns_event_fw_loop () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #4 0x00007f5bd4d33ac9 in event_loop_thread_func () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 33
>>>> (Thread 0x7f5bb2b84700 (LWP 15559)):
>>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>>>> #1 0x00007f5bd4acc599 in DS_Sleep () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007f5bc3e41e77 in deadlock_threadmain () at
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #4 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #5 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 32
>>>> (Thread 0x7f5bb2383700 (LWP 15560)):
>>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>>>> #1 0x00007f5bd4acc599 in DS_Sleep () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007f5bc3e45a87 in checkpoint_threadmain () at
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #4 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #5 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 31
>>>> (Thread 0x7f5bb1b82700 (LWP 15561)):
>>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>>>> #1 0x00007f5bd4acc599 in DS_Sleep () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007f5bc3e420c7 in trickle_threadmain () at
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #4 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #5 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 30
>>>> (Thread 0x7f5bb1381700 (LWP 15562)):
>>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>>>> #1 0x00007f5bd4acc599 in DS_Sleep () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007f5bc3e3ce47 in perf_threadmain () at
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #4 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #5 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 29
>>>> (Thread 0x7f5bb0b80700 (LWP 15563)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #3 0x00007f5bc920213e in cos_cache_wait_on_change () at
>>>> /usr/lib64/dirsrv/plugins/libcos-plugin.so
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 28
>>>> (Thread 0x7f5bb037f700 (LWP 15564)):
>>>> #0 0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
>>>> #1 0x00007f5bc6d7e44c in ipa_cldap_worker () at
>>>> /usr/lib64/dirsrv/plugins/libipa_cldap.so
>>>> #2 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #3 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 27
>>>> (Thread 0x7f5bd5132700 (LWP 15566)):
>>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #3 0x00007f5bc3572033 in _cl5TrimMain () at
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 26
>>>> (Thread 0x7f5bafb7e700 (LWP 15567)):
>>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #3 0x00007f5bc3587fa4 in protocol_sleep () at
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #4 0x00007f5bc358b75a in repl5_inc_run () at
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #5 0x00007f5bc358fbec in prot_thread_main () at
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #6 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #7 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #8 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 25
>>>> (Thread 0x7f5baf37d700 (LWP 15568)):
>>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #3 0x00007f5bc3587fa4 in protocol_sleep () at
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #4 0x00007f5bc358b75a in repl5_inc_run () at
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #5 0x00007f5bc358fbec in prot_thread_main () at
>>>> /usr/lib64/dirsrv/plugins/libreplication-plugin.so
>>>> #6 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #7 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #8 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 24
>>>> (Thread 0x7f5bae759700 (LWP 15569)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #3 0x00007f5bc22d11ad in roles_cache_wait_on_change () at
>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 23
>>>> (Thread 0x7f5badf58700 (LWP 15570)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #3 0x00007f5bc22d11ad in roles_cache_wait_on_change () at
>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 22
>>>> (Thread 0x7f5bad757700 (LWP 15571)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd4ab8eb8 in slapi_wait_condvar () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #3 0x00007f5bc22d11ad in roles_cache_wait_on_change () at
>>>> /usr/lib64/dirsrv/plugins/libroles-plugin.so
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 21
>>>> (Thread 0x7f5bacd4f700 (LWP 15572)):
>>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #3 0x0000564591e10b08 in housecleaning ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 20
>>>> (Thread 0x7f5bac54e700 (LWP 15573)):
>>>> #0 0x00007f5bd21fccf2 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2852d27 in pt_TimedWait () at /lib64/libnspr4.so
>>>> #2 0x00007f5bd285320e in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #3 0x00007f5bd4a5ba63 in eq_loop () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 19
>>>> (Thread 0x7f5babd4d700 (LWP 15574)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 18
>>>> (Thread 0x7f5bab54c700 (LWP 15575)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 17
>>>> (Thread 0x7f5baad4b700 (LWP 15576)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 16
>>>> (Thread 0x7f5baa54a700 (LWP 15577)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 15
>>>> (Thread 0x7f5ba9d49700 (LWP 15578)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 14
>>>> (Thread 0x7f5ba9548700 (LWP 15579)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 13
>>>> (Thread 0x7f5ba8d47700 (LWP 15580)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 12
>>>> (Thread 0x7f5ba8546700 (LWP 15581)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 11
>>>> (Thread 0x7f5ba7d45700 (LWP 15582)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 10
>>>> (Thread 0x7f5ba7544700 (LWP 15583)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 9
>>>> (Thread 0x7f5ba6d43700 (LWP 15584)):
>>>> #0 0x00007f5bd189ae9d in poll () at /lib64/libc.so.6
>>>> #1 0x00007f5bd2854bc7 in _pr_poll_with_poll () at /lib64/libnspr4.so
>>>> #2 0x0000564591e07049 in connection_read_operation ()
>>>> #3 0x0000564591e0765d in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 8
>>>> (Thread 0x7f5ba6542700 (LWP 15585)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 7
>>>> (Thread 0x7f5ba5d41700 (LWP 15586)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 6
>>>> (Thread 0x7f5ba5540700 (LWP 15587)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 5
>>>> (Thread 0x7f5ba4d3f700 (LWP 15588)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 4
>>>> (Thread 0x7f5ba453e700 (LWP 15589)):
>>>> #0 0x00007f5bd21fc945 in pthread_cond_wait@(a)GLIBC_2.3.2 () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd2853290 in PR_WaitCondVar () at /lib64/libnspr4.so
>>>> #2 0x0000564591e05fce in [IDLE THREAD] connection_wait_for_new_work
>>>> ()
>>>> #3 0x0000564591e075d7 in connection_threadmain ()
>>>> #4 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #5 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #6 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 3
>>>> (Thread 0x7f5ba3900700 (LWP 15590)):
>>>> #0 0x00007f5bd189cc03 in select () at /lib64/libc.so.6
>>>> #1 0x00007f5bd4acc599 in DS_Sleep () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007f5bc1ea5c6e in backend_set_entry_from () at
>>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>>>> #3 0x00007f5bc1ea8906 in backend_shr_set_entry_cb () at
>>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>>>> #4 0x00007f5bd4aac614 in send_ldap_search_entry_ext () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007f5bd4aace0c in send_ldap_search_entry () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #6 0x00007f5bd4a87ca4 in send_entry.isra.0 () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #7 0x00007f5bd4a88208 in send_results_ext.constprop.4 () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #8 0x00007f5bd4a8a0d1 in op_shared_search () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #9 0x00007f5bd4a9b507 in search_internal_callback_pb () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #10 0x00007f5bc1ead018 in backend_shr_set_config_entry_add () at
>>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>>>> #11 0x00007f5bc1ea7a2d in backend_set_config_entry_add_cb () at
>>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>>>> #12 0x00007f5bc1ea8ac1 in backend_shr_data_initialize_thread_cb () at
>>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>>>> #13 0x00007f5bc1eba597 in wrap_pthread_starter () at
>>>> /usr/lib64/dirsrv/plugins/schemacompat-plugin.so
>>>> #14 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #15 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #16 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 2
>>>> (Thread 0x7f5ba30ff700 (LWP 15606)):
>>>> #0 0x00007f5bd21fc2ae in pthread_rwlock_wrlock () at
>>>> /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4a7f0e5 in slapi_mtn_be_set_readonly () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #2 0x00007f5bc3e921ac in instance_set_busy_and_readonly () at
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #3 0x00007f5bc3e8d5d1 in ldbm_back_ldbm2index () at
>>>> /usr/lib64/dirsrv/plugins/libback-ldbm.so
>>>> #4 0x00007f5bd4ac5942 in task_index_thread () at
>>>> /usr/lib64/dirsrv/libslapd.so.0
>>>> #5 0x00007f5bd2858c8b in _pt_root () at /lib64/libnspr4.so
>>>> #6 0x00007f5bd21f8dd5 in start_thread () at /lib64/libpthread.so.0
>>>> #7 0x00007f5bd18a5b3d in clone () at /lib64/libc.so.6 Thread 1
>>>> (Thread 0x7f5bd5135940 (LWP 15536)):
>>>> #0 0x00007f5bd21f9f47 in pthread_join () at /lib64/libpthread.so.0
>>>> #1 0x00007f5bd4d34b2f in ns_thrpool_wait () at
>>>> /usr/lib64/dirsrv/libnunc-stans.so.0
>>>> #2 0x0000564591e0d862 in slapd_daemon ()
>>>> #3 0x0000564591dfe86f in main ()
>>>>
>>>> Thanks & Regards.
>>>> ______________________________
>>>>
>>>> -----Original Message-----
>>>> From: thierry bordaz <tbordaz(a)redhat.com>
>>>> Sent: Thursday, May 03, 2018 9:31
>>>> To: Rob Crittenden <rcritten(a)redhat.com>; FreeIPA users list
>>>> <freeipa-users(a)lists.fedorahosted.org>; SOLER SANGUESA Miguel
>>>> <solerm(a)unicc.org>
>>>> Cc: Ludwig Krispenz <lkrispen(a)redhat.com>
>>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating from
>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>>>>
>>>> Hi,
>>>>
>>>> During indexing task we should see in the task status the periodic
>>>> progression of the indexing.
>>>> May be the indexing is hanging somewhere. When the problem occurs
>>>> could you provide a pstack of the dirsrv server ?
>>>>
>>>> best regards
>>>> thierry
>>>>
>>>> On 05/02/2018 10:27 PM, Rob Crittenden wrote:
>>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
>>>>>> Hello,
>>>>>>
>>>>>> This is the output of the command (seems that is not complete):
>>>>>>
>>>>>> # ldapsearch -H
ldapi://%2fvar%2frun%2fslapd-IPA-EXAMPLE-ORG.socket
>>>>>> -b
>>>>>>
cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
>>>>>> n
>>>>>> =config
>>>>>> -s base
>>>>>> SASL/EXTERNAL authentication started SASL username:
>>>>>> gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth
>>>>>> SASL SSF: 0
>>>>>> # extended LDIF
>>>>>> #
>>>>>> # LDAPv3
>>>>>> # base
>>>>>>
<cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
>>>>>> c
>>>>>> n=config>
>>>>>> with scope baseObject
>>>>>> # filter: (objectclass=*)
>>>>>> # requesting: ALL
>>>>>> #
>>>>>>
>>>>>> # indextask_description_137444551994158920_5958, index, tasks,
>>>>>> config
>>>>>> dn:
>>>>>>
cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,c
>>>>>> n
>>>>>> =conf
>>>>>> ig
>>>>>> objectClass: top
>>>>>> objectClass: extensibleObject
>>>>>> nsindexattribute: description
>>>>>> nsinstance: userRoot
>>>>>> cn: indextask_description_137444551994158920_5958
>>>>>> nstaskcurrentitem: 0
>>>>>> nstasktotalitems: 1
>>>>> IPA is looking for nstaskstatus and since that isn't there it is
>>>>> looping forever.
>>>>>
>>>>> I've cc'd a couple of the 389-ds devs to see what they think
about
>>>>> it.
>>>>>
>>>>> rob
>>>>>
>>>>>> # search result
>>>>>> search: 3
>>>>>> result: 0 Success
>>>>>>
>>>>>> # numResponses: 2
>>>>>> # numEntries: 1
>>>>>>
>>>>>> Thanks & Regards.
>>>>>> ______________________________
>>>>>> -----Original Message-----
>>>>>> From: Rob Crittenden <rcritten(a)redhat.com>
>>>>>> Sent: Tuesday, May 01, 2018 15:18
>>>>>> To: FreeIPA users list
<freeipa-users(a)lists.fedorahosted.org>
>>>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org>
>>>>>> Subject: Re: [Freeipa-users] Re: Problem on dirsrv when updating
>>>>>> from
>>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>>>>>>
>>>>>> SOLER SANGUESA Miguel via FreeIPA-users wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> Thank you for your answer, now dirsrv can start, but after
running
>>>>>>> the "
>>>>>>> dirsrv(a)IPA-UNICC-ORG.service " it hungs:
>>>>>>>
>>>>>>> # ipa-server-upgrade
>>>>>>>
>>>>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>>>>>>>
>>>>>>> [1/8]: saving configuration
>>>>>>>
>>>>>>> [2/8]: disabling listeners
>>>>>>>
>>>>>>> [3/8]: enabling DS global lock
>>>>>>>
>>>>>>> [4/8]: starting directory server
>>>>>>>
>>>>>>> [5/8]: updating schema
>>>>>>>
>>>>>>> [6/8]: upgrading server
>>>>>>>
>>>>>>> The latest messages on the log is (there are no logs on the
>>>>>>> lasts 2h):
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG
>>>>>>> ---------------------------------------------
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG Final value after applying
updates
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG dn:
>>>>>>> cn=description,cn=index,cn=userroot,cn=ldbm
>>>>>>> database,cn=plugins,cn=config
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG objectclass:
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG top
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG nsindex
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG nsindextype:
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG eq
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG sub
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG cn:
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG description
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG nssystemindex:
>>>>>>>
>>>>>>> 2018-05-01T08:13:14Z DEBUG false
>>>>>>>
>>>>>>> 2018-05-01T08:13:19Z DEBUG Creating task to index attribute:
>>>>>>> description
>>>>>>>
>>>>>>> 2018-05-01T08:13:19Z DEBUG Task id:
>>>>>>>
cn=indextask_description_137444551994158920_5958,cn=index,cn=tasks,
>>>>>>> c
>>>>>>> n=
>>>>>>> config
>>>>>>>
>>>>>>> If I check the ipa services, dirsrv is the only one running:
>>>>>>>
>>>>>>> # ipactl status
>>>>>>>
>>>>>>> Directory Service: RUNNING
>>>>>>>
>>>>>>> krb5kdc Service: STOPPED
>>>>>>>
>>>>>>> kadmin Service: STOPPED
>>>>>>>
>>>>>>> named Service: STOPPED
>>>>>>>
>>>>>>> httpd Service: STOPPED
>>>>>>>
>>>>>>> ipa-custodia Service: STOPPED
>>>>>>>
>>>>>>> ntpd Service: RUNNING
>>>>>>>
>>>>>>> pki-tomcatd Service: STOPPED
>>>>>>>
>>>>>>> smb Service: STOPPED
>>>>>>>
>>>>>>> winbind Service: STOPPED
>>>>>>>
>>>>>>> ipa-otpd Service: STOPPED
>>>>>>>
>>>>>>> ipa-dnskeysyncd Service: STOPPED
>>>>>>>
>>>>>>> ipa: INFO: The ipactl command was successful
>>>>>>>
>>>>>>> Should I stop the ipa-server-upgrade and start ipa services?
>>>>>> It depends on how many entries you have. This is an indexing
task
>>>>>> and while 2 hours seems a bit excessive, it depends on how much
work
>>>>>> it has to do.
>>>>>>
>>>>>> The 389-ds access and/or error logs may provide details.
>>>>>>
>>>>>> rob
>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Alexander Bokovoy <abokovoy(a)redhat.com>
>>>>>>> Sent: Tuesday, May 01, 2018 9:56
>>>>>>> To: FreeIPA users list
<freeipa-users(a)lists.fedorahosted.org>
>>>>>>> Cc: SOLER SANGUESA Miguel <solerm(a)unicc.org>
>>>>>>> Subject: Re: [Freeipa-users] Problem on dirsrv when updating
from
>>>>>>> 4.5.0 (RHEL 7.4) to 4.5.4 (RHEL 7.5)
>>>>>>>
>>>>>>> On ti, 01 touko 2018, SOLER SANGUESA Miguel via FreeIPA-users
>>>>>>> wrote:
>>>>>>>
>>>>>>> >hello,
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> >I have an IPA master that updated from 4.5.0 (RHEL
7.4) to
>>>>>>> 4.5.4 (RHEL 7.5). An hour later I tried to do the same with
the
>>>>>>> unique replica I have, but after update dirsrv is not
starting.
>>>>>>>
>>>>>>> >It says it is needed run
"ipa-server-upgrade", but it also
>>>>>>> fails:
>>>>>>>
>>>>>>> ># ipactl start
>>>>>>>
>>>>>>> >Upgrade required: please run ipa-server-upgrade
command
>>>>>>> Aborting ipactl
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> ># ipa-server-upgrade
>>>>>>>
>>>>>>> >Upgrading IPA:. Estimated time: 1 minute 30 seconds
>>>>>>>
>>>>>>> > [1/8]: saving configuration
>>>>>>>
>>>>>>> > [2/8]: disabling listeners
>>>>>>>
>>>>>>> > [3/8]: enabling DS global lock
>>>>>>>
>>>>>>> > [4/8]: starting directory server
>>>>>>>
>>>>>>> > [error] CalledProcessError: Command
'/bin/systemctl start
>>>>>>>
>>>>>>> >dirsrv(a)IPA-EXAMOLE-ORG.service
>>>>>>> <mailto:dirsrv@IPA-EXAMOLE-ORG.service>' returned
non-zero exit
>>>>>>> status
>>>>>>> 1
>>>>>>>
>>>>>>> > [cleanup]: stopping directory server
>>>>>>>
>>>>>>> > [cleanup]: restoring configuration
>>>>>>>
>>>>>>> >IPA server upgrade failed: Inspect
/var/log/ipaupgrade.log and
>>>>>>> run command ipa-server-upgrade manually.
>>>>>>>
>>>>>>> >Unexpected error - see /var/log/ipaupgrade.log for
details:
>>>>>>>
>>>>>>> >CalledProcessError: Command '/bin/systemctl
start
>>>>>>>
>>>>>>> >dirsrv(a)IPA-EXAMPLE-ORG.service
>>>>>>> <mailto:dirsrv@IPA-EXAMPLE-ORG.service>' returned
non-zero exit
>>>>>>> status
>>>>>>> 1 The
>>>>>>>
>>>>>>> >ipa-server-upgrade command failed. See
/var/log/ipaupgrade.log
>>>>>>> for more
>>>>>>>
>>>>>>> >information
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> >On the log I can see:
>>>>>>>
>>>>>>> >2018-04-30T14:36:15Z DEBUG Starting external process
>>>>>>>
>>>>>>> >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl
is-active
>>>>>>>
>>>>>>> >dirsrv(a)IPA-EXAMPLE-ORG.service
>>>>>>> <mailto:dirsrv@IPA-EXAMPLE-ORG.service>
>>>>>>> 2018-04-30T14:36:15Z DEBUG Process
>>>>>>>
>>>>>>> >finished, return code=3 2018-04-30T14:36:15Z DEBUG
>>>>>>> stdout=failed ...
>>>>>>>
>>>>>>> >2018-04-30T14:36:15Z DEBUG [4/8]: starting
directory server
>>>>>>>
>>>>>>> >2018-04-30T14:36:15Z DEBUG Starting external process
>>>>>>>
>>>>>>> >2018-04-30T14:36:15Z DEBUG args=/bin/systemctl start
>>>>>>>
>>>>>>> >dirsrv(a)IPA-EXAMPLE-ORG.service
>>>>>>> <mailto:dirsrv@IPA-EXAMPLE-ORG.service>
>>>>>>> 2018-04-30T14:36:15Z DEBUG Process
>>>>>>>
>>>>>>> >finished, return code=1 2018-04-30T14:36:15Z DEBUG
stdout=
>>>>>>>
>>>>>>> >2018-04-30T14:36:15Z DEBUG stderr=Job for
>>>>>>> dirsrv(a)IPA-EXAMPLE-ORG.service
>>>>>>> <mailto:dirsrv@IPA-EXAMPLE-ORG.service>
>>>>>>> failed because the control process exited with error code.
See
>>>>>>> "systemctl status dirsrv(a)IPA-EXAMPLE-ORG.service
>>>>>>> <mailto:dirsrv@IPA-EXAMPLE-ORG.service>" and
"journalctl -xe" for
>>>>>>> details.
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> >2018-04-30T14:36:15Z DEBUG Traceback (most recent
call last):
>>>>>>>
>>>>>>> > File
>>>>>>>
"/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>>>>>>> line 504, in start_creation
>>>>>>>
>>>>>>> > run_step(full_msg, method)
>>>>>>>
>>>>>>> > File
>>>>>>>
"/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
>>>>>>> line 494, in run_step
>>>>>>>
>>>>>>> > method()
>>>>>>>
>>>>>>> > File
>>>>>>>
"/usr/lib/python2.7/site-packages/ipaserver/install/upgradeinstance.
>>>>>>>
>>>>>>> py
>>>>>>> ",
>>>>>>> line 95, in __start
>>>>>>>
>>>>>>> > srv.start(self.serverid, ldapi=True)
>>>>>>>
>>>>>>> > File
>>>>>>>
"/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py",
>>>>>>> line 161, in start
>>>>>>>
>>>>>>> > instance_name, capture_output=capture_output,
wait=wait)
>>>>>>>
>>>>>>> > File
>>>>>>>
"/usr/lib/python2.7/site-packages/ipaplatform/base/services.py",
>>>>>>> line 294, in start
>>>>>>>
>>>>>>> > skip_output=not capture_output)
>>>>>>>
>>>>>>> > File
>>>>>>>
"/usr/lib/python2.7/site-packages/ipapython/ipautil.py",
>>>>>>> line 542, in run
>>>>>>>
>>>>>>> > raise CalledProcessError(p.returncode,
arg_string,
>>>>>>> str(output))
>>>>>>>
>>>>>>> >CalledProcessError: Command '/bin/systemctl
start
>>>>>>>
>>>>>>> >dirsrv(a)IPA-EXAMPLE-ORG.service
>>>>>>> <mailto:dirsrv@IPA-EXAMPLE-ORG.service>' returned
non-zero exit
>>>>>>> status
>>>>>>> 1
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> >2018-04-30T14:36:15Z DEBUG [error]
CalledProcessError:
>>>>>>> Command '/bin/systemctl start
dirsrv(a)IPA-EXAMPLE-ORG.service
>>>>>>> <mailto:dirsrv@IPA-EXAMPLE-ORG.service>' returned
non-zero exit
>>>>>>> status
>>>>>>> 1
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> >Checking /var/log/dirsrv/slapd-IPA-EXAMPLE-ORG/errors
I show:
>>>>>>>
>>>>>>> >[30/Apr/2018:16:04:52.584220922 +0200] - ERR -
>>>>>>> slapd_bootstrap_config
>>>>>>> - The default password storage scheme could not be read or
was not
>>>>>>> found in the file /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse.ldif.
It is
>>>>>>> mandatory.
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> >Checking on internet I show that "dse.ldif"
could be
>>>>>>> corrupted,
>>>>>>> so I changed with "dse.ldif.startOK" without any
change and then I
>>>>>>> changed with "dse.ldif.bak". The problem persist
but the error
>>>>>>> has changed:
>>>>>>>
>>>>>>> >[30/Apr/2018:16:32:13.435210918 +0200] - NOTICE -
>>>>>>> config_set_port
>>>>>>> -
>>>>>>>
>>>>>>> >Non-Secure Port Disabled
>>>>>>>
>>>>>>> >[30/Apr/2018:16:32:13.556581301 +0200] - ERR -
>>>>>>> symload_report_error
>>>>>>> -
>>>>>>>
>>>>>>> >Netscape Portable Runtime error -5975:
>>>>>>>
>>>>>>> >/usr/lib64/dirsrv/plugins/libreplication-plugin.so:
undefined
>>>>>>> symbol:
>>>>>>>
>>>>>>> >replication_legacy_plugin_init
>>>>>>>
>>>>>>> >[30/Apr/2018:16:32:13.561590553 +0200] - ERR -
>>>>>>> symload_report_error
>>>>>>> -
>>>>>>>
>>>>>>> >Could not load symbol
"replication_legacy_plugin_init" from
>>>>>>>
>>>>>>>
>"/usr/lib64/dirsrv/plugins/libreplication-plugin.so" for plugin
>>>>>>> Legacy
>>>>>>>
>>>>>>> >Replication Plugin
>>>>>>>
>>>>>>> >[30/Apr/2018:16:32:13.564590264 +0200] - ERR -
>>>>>>> load_plugin_entry
>>>>>>> - Unable to load plugin "cn=Legacy Replication
>>>>>>> Plugin,cn=plugins,cn=config"
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> >I saw a bug about this problem, but it is still
opened:
>>>>>>>
>>>>>>>>
https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=152944
>>>>>>>> 2
>>>>>>> >
>>>>>>>
>>>>>>> >Any idea how to fix the issue?
>>>>>>>
>>>>>>> >
>>>>>>>
>>>>>>> >If it is not possible to fix it, can I remove the
replica from
>>>>>>> IPA and install it again with the same name?
>>>>>>>
>>>>>>> A quick fix could be to remove an entry for cn=Legacy
Replication
>>>>>>> Plugin,cn=plugins,cn=config from
>>>>>>> /etc/dirsrv/slapd-IPA-EXAMPLE-ORG/dse/ldif
>>>>>>>
>>>>>>> when dirsrv is down.
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> / Alexander Bokovoy
>>>>>>>
>>>>>>> Sr. Principal Software Engineer
>>>>>>>
>>>>>>> Security / Identity Management Engineering Red Hat Limited,
Finland
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>>> To unsubscribe send an email to
>>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>> To unsubscribe send an email to
>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>
>>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org