PR_ASSERT problem
by Carsten Grzemba
my build on OpenIndiana with the IllumOS provided NSPR cores on startup of some commands:
$ pwdhash
Assertion failure: 0 == lock->notified.length, at /jenkins/jobs/oi-userland/workspace/components/library/mozilla-nspr/nspr-4.12/nspr/pr/src/pthreads/ptsynch.c:159
/usr/bin/pwdhash: line 60: 5993: Abort(coredump)
$ pstack core
core 'core' of 5993: /usr/bin/amd64/pwdhash-bin
fffffd7fff2b4eda _lwp_kill () + a
fffffd7fff2480e0 raise (6) + 20
fffffd7fff220818 abort () + 98
fffffd7ff943fda2 ???????? ()
fffffd7ff9457828 PR_DestroyLock () + d8
fffffd7ff978bc2a slapi_entry_free () + 11a
0000000000402888 init_config () + 2d8
0000000000402d43 main () + 353
000000000040235c _start () + 6c
Before I dive in the source code, has anyone a hint where I can start to search what happens here?
<signaturebeforequotedtext></signaturebeforequotedtext><signatureafterquotedtext></signatureafterquotedtext>
6 years, 7 months
Build failed in Jenkins: NIGHTLY #92
by mareynol@redhat.com
See <http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/92/display/...>
------------------------------------------
[...truncated 6809 lines...]
File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
self.run()
File "<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/3...",> line 51, in run
conn = self.openConnection(self.inst)
File "<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/3...",> line 45, in openConnection
server.open()
File "<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...",> line 1127, in open
raise e
SERVER_DOWN: {'\''desc'\'': "Can'\''t contact LDAP server"}
INFO:dirsrvtests.tests.suites.replication.cleanallruv_test:test_stress_clean: put master 4 into read-only mode...
CRITICAL:dirsrvtests.tests.suites.replication.cleanallruv_test:test_stress_clean: Failed to put master 4 into read-only mode: error Can'\''t contact LDAP server
________________________ test_multiple_tasks_with_force ________________________
topology_m4 = <lib389.topologies.TopologyMain object at 0x7fd2beb69f50>
def test_multiple_tasks_with_force(topology_m4):
"""Check that multiple tasks with a '\''force'\'' option work properly
:ID: eb76a93d-8d1c-405e-9f25-6e8d5a781098
:feature: CleanAllRUV
:setup: Replication setup with four masters
:steps: 1. Stop master 3
2. Add a bunch of updates to master 4
3. Disable replication on master 4
4. Start master 3
5. Remove agreements to master 4 from other masters
6. Run a cleanallruv task on master 1 with a '\''force'\'' option '\''on'\''
7. Run one more cleanallruv task on master 1 with a '\''force'\'' option '\''off'\''
8. Check that everything was cleaned and no hanging tasks left
:expectedresults: Everything was cleaned and no hanging tasks left
"""
log.info('\''Running test_multiple_tasks_with_force...'\'')
# Stop master 3, while we update master 4, so that 3 is behind the other masters
topology_m4.ms["master3"].stop(timeout=10)
# Add a bunch of updates to master 4
m4_add_users = AddUsers(topology_m4.ms["master4"], 1500)
m4_add_users.start()
m4_add_users.join()
# Disable master 4
log.info('\''test_multiple_tasks_with_force: disable master 4...'\'')
> topology_m4.ms["master4"].replica.disableReplication(DEFAULT_SUFFIX)
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/3...>:817:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...>:453: in disableReplication
mtents = self.conn.mappingtree.list(suffix=nsuffix)
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...>:70: in list
filt)
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...>:162: in inner
return f(*args, **kwargs)
/usr/lib64/python2.7/site-packages/ldap/ldapobject.py:597: in search_s
return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...>:162: in inner
return f(*args, **kwargs)
/usr/lib64/python2.7/site-packages/ldap/ldapobject.py:590: in search_ext_s
msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...>:162: in inner
return f(*args, **kwargs)
/usr/lib64/python2.7/site-packages/ldap/ldapobject.py:586: in search_ext
timeout,sizelimit,
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...>:162: in inner
return f(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <lib389.DirSrv object at 0x7fd2bf00c650>
func = <built-in method search_ext of LDAP object at 0x7fd2c8c57e18>
args = ('\''cn=mapping tree,cn=config'\'', 1, '\''(cn=dc=example,dc=com)'\'', None, 0, None, ...)
kwargs = {}, diagnostic_message_success = None
e = SERVER_DOWN({'\''desc'\'': "Can'\''t contact LDAP server"},)
def _ldap_call(self,func,*args,**kwargs):
"""
Wrapper method mainly for serializing calls into OpenLDAP libs
and trace logs
"""
self._ldap_object_lock.acquire()
if __debug__:
if self._trace_level>=1:
self._trace_file.write('\''*** %s %s - %s\n%s\n'\'' % (
repr(self),
self._uri,
'\''.'\''.join((self.__class__.__name__,func.__name__)),
pprint.pformat((args,kwargs))
))
if self._trace_level>=9:
traceback.print_stack(limit=self._trace_stack_limit,file=self._trace_file)
diagnostic_message_success = None
try:
try:
> result = func(*args,**kwargs)
E SERVER_DOWN: {'\''desc'\'': "Can'\''t contact LDAP server"}
/usr/lib64/python2.7/site-packages/ldap/ldapobject.py:106: SERVER_DOWN
----------------------------- Captured stderr call -----------------------------
INFO:dirsrvtests.tests.suites.replication.cleanallruv_test:Running test_multiple_tasks_with_force...
Exception in thread Thread-67:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 804, in __bootstrap_inner
self.run()
File "<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/3...",> line 51, in run
conn = self.openConnection(self.inst)
File "<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/3...",> line 45, in openConnection
server.open()
File "<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...",> line 1127, in open
raise e
SERVER_DOWN: {'\''desc'\'': "Can'\''t contact LDAP server"}
INFO:dirsrvtests.tests.suites.replication.cleanallruv_test:test_multiple_tasks_with_force: disable master 4...
_____________________________ test_mail_attr_repl ______________________________
topo_r = <lib389.topologies.TopologyMain object at 0x7fd2beb69b50>
test_user = None
def test_mail_attr_repl(topo_r, test_user):
"""Check that no crash happens during mail attribute replication
:ID: 959edc84-05be-4bf9-a541-53afae482052
:feature: Single master replication
:setup: Replication setup with master and consumer instances,
test user on master
:steps: 1. Check that user was replicated to consumer
2. Back up mail database file
3. Remove mail attribute from the user entry
4. Restore mail database
5. Search for the entry with a substring '\''mail=user*'\''
6. Search for the entry once again to make sure that server is alive
:expectedresults: No crash happens
"""
master = topo_r.ms["master1"]
consumer = topo_r.cs["consumer1"]
log.info("Wait for a user to be replicated")
time.sleep(3)
log.info("Check that replication is working")
entries = consumer.search_s(DEFAULT_SUFFIX, ldap.SCOPE_SUBTREE, "uid={}".format(TEST_USER_NAME),
["uid"])
assert entries, "User {} wasn'\''t replicated successfully".format(TEST_USER_NAME)
entries = consumer.backend.list(DEFAULT_SUFFIX)
db_dir = entries[0]["nsslapd-directory"]
mail_db = filter(lambda fl: fl.startswith("mail"), os.listdir(db_dir))
assert mail_db, "mail.* wasn'\''t found in {}"
mail_db_path = os.path.join(db_dir, mail_db[0])
backup_path = os.path.join(DEFAULT_BACKUPDIR, mail_db[0])
consumer.stop()
log.info("Back up {} to {}".format(mail_db_path, backup_path))
shutil.copyfile(mail_db_path, backup_path)
consumer.start()
log.info("Remove '\''mail'\'' attr from master")
try:
master.modify_s(TEST_USER_DN, [(ldap.MOD_DELETE, '\''mail'\'', '\''{}(a)redhat.com'\''.format(TEST_USER_NAME))])
except ldap.LDAPError as e:
log.error('\''Failed to delete att user {}: error {}'\''.format(TEST_USER_DN, e.message['\''desc'\'']))
raise e
log.info("Wait for the replication to happen")
time.sleep(5)
consumer.stop()
log.info("Restore {} to {}".format(backup_path, mail_db_path))
shutil.copyfile(backup_path, mail_db_path)
> consumer.start()
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/3...>:133:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
<http://vm-166.abc.idm.lab.eng.brq.redhat.com:8080/job/NIGHTLY/ws/source/l...>:1180: in start
"dirsrv@%s" % self.serverid])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
popenargs = (['\''/usr/bin/systemctl'\'', '\''start'\'', '\''dirsrv@consumer1'\''],), kwargs = {}
retcode = 1, cmd = ['\''/usr/bin/systemctl'\'', '\''start'\'', '\''dirsrv@consumer1'\'']
def check_call(*popenargs, **kwargs):
"""Run command with arguments. Wait for command to complete. If
the exit code was zero then return, otherwise raise
CalledProcessError. The CalledProcessError object will have the
return code in the returncode attribute.
The arguments are the same as for the Popen constructor. Example:
check_call(["ls", "-l"])
"""
retcode = call(*popenargs, **kwargs)
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
> raise CalledProcessError(retcode, cmd)
E CalledProcessError: Command '\''['\''/usr/bin/systemctl'\'', '\''start'\'', '\''dirsrv@consumer1'\'']'\'' returned non-zero exit status 1
/usr/lib64/python2.7/subprocess.py:186: CalledProcessError
---------------------------- Captured stdout setup -----------------------------
OK group dirsrv exists
OK user dirsrv exists
OK group dirsrv exists
OK user dirsrv exists
('\''Update succeeded: status '\'', '\''0 Total update succeeded'\'')
---------------------------- Captured stderr setup -----------------------------
INFO:lib389.topologies:Instance with parameters {'\''ldap-port'\'': 39201, '\''suffix'\'': '\''dc=example,dc=com'\'', '\''krb5_realm'\'': None, '\''deployed-dir'\'': '\''/usr'\'', '\''inst-backupdir'\'': '\''/tmp'\'', '\''hostname'\'': '\''localhost'\'', '\''server-id'\'': '\''consumer1'\'', '\''root-pw'\'': '\''password'\'', '\''root-dn'\'': '\''cn=Directory Manager'\'', '\''group-id'\'': None, '\''InstScriptsEnabled'\'': None, '\''user-id'\'': None, '\''ldap-secureport'\'': None} was created.
INFO:lib389:Found entry dn: cn=replrepl,cn=config
cn: bind dn pseudo user
cn: replrepl
objectClass: top
objectClass: person
sn: bind dn pseudo user
userPassword: {SSHA512}Su2JJHV1gnmfuj9r4aBSVuhCu+cBCPkbrsrZ5qvNy8YjloaOOb6ygdaSDEbtTwYvRoMiyS8WUTK/YqDHKV2GM0jIWOYHq3Mj
INFO:lib389.topologies:Instance with parameters {'\''ldap-port'\'': 39001, '\''suffix'\'': '\''dc=example,dc=com'\'', '\''krb5_realm'\'': None, '\''deployed-dir'\'': '\''/usr'\'', '\''inst-backupdir'\'': '\''/tmp'\'', '\''hostname'\'': '\''localhost'\'', '\''server-id'\'': '\''master1'\'', '\''root-pw'\'': '\''password'\'', '\''root-dn'\'': '\''cn=Directory Manager'\'', '\''group-id'\'': None, '\''InstScriptsEnabled'\'': None, '\''user-id'\'': None, '\''ldap-secureport'\'': None} was created.
INFO:lib389:Found entry dn: cn=replrepl,cn=config
cn: bind dn pseudo user
cn: replrepl
objectClass: top
objectClass: person
sn: bind dn pseudo user
userPassword: {SSHA512}lWE9XevV1srQKk05+U6I78KeAgCA6+os9FtH7CkhHjlKu+cO3fJZ22EJ1lSZezrz+r74GtTRS+EF9PQg26cNRPnP1moqg9nI
INFO:Replica:Starting async replication cn=meTo_localhost:39201,cn=replica,cn=dc\3Dexample\2Cdc\3Dcom,cn=mapping tree,cn=config
INFO:dirsrvtests.tests.suites.replication.single_master_test:Adding user uid=smrepl_test,dc=example,dc=com
----------------------------- Captured stderr call -----------------------------
INFO:dirsrvtests.tests.suites.replication.single_master_test:Wait for a user to be replicated
INFO:dirsrvtests.tests.suites.replication.single_master_test:Check that replication is working
INFO:lib389:List backend with suffix=dc=example,dc=com
INFO:dirsrvtests.tests.suites.replication.single_master_test:Back up /var/lib/dirsrv/slapd-consumer1/db/userRoot/mail.db to /tmp/mail.db
INFO:dirsrvtests.tests.suites.replication.single_master_test:Remove '\''mail'\'' attr from master
INFO:dirsrvtests.tests.suites.replication.single_master_test:Wait for the replication to happen
INFO:dirsrvtests.tests.suites.replication.single_master_test:Restore /tmp/mail.db to /var/lib/dirsrv/slapd-consumer1/db/userRoot/mail.db
Job for dirsrv(a)consumer1.service failed because a fatal signal was delivered to the control process.
See "systemctl status dirsrv(a)consumer1.service" and "journalctl -xe" for details.
======== 10 failed, 573 passed, 1 skipped, 33 error in 11631.25 seconds ========'
+ '[' 1 -ne 0 ']'
+ echo CI Tests 'FAILED!'
CI Tests FAILED!
+ MSG=FAILED
+ RC=1
+ sudo /usr/sbin/sendmail mreynolds(a)redhat.com firstyear(a)redhat.com
+ sudo rm -rf /var/tmp/slapd.vg.51356 /var/tmp/slapd.vg.78052 /var/tmp/slapd.vg.78194 /var/tmp/slapd.vg.95273
+ exit 1
Build step 'Execute shell' marked build as failure
6 years, 7 months