On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
> On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
>
>> On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
>>> On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
>>>> On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
>>>>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
>>>>>> On 08/02/2017 01:43 AM, Ian Harding wrote:
>>>>>>> On 08/01/2017 12:03 PM, Rob Crittenden wrote:
>>>>>>>> Ian Harding wrote:
>>>>>>>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
>>>>>>>>>> On 08/01/2017 03:11 PM, Ian Harding wrote:
>>>>>>>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud
wrote:
>>>>>>>>>>>> On 08/01/2017 01:32 AM, Ian Harding via
FreeIPA-users wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 07/31/2017 11:34 AM, Rob
Crittenden wrote:
>>>>>>>>>>>>>> Ian Harding via FreeIPA-users
wrote:
>>>>>>>>>>>>>>> I had an unexpected restart
of an IPA server that had
>>>>>>>>>>>>>>> apparently had
>>>>>>>>>>>>>>> updates run but had not been
restarted. ipactl says
>>>>>>>>>>>>>>> pki-tomcatd
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>> not start.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Strangely, the actual service
appears to be running:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> dogtag is an application within
tomcat so tomcat can run
>>>>>>>>>>>>>> without
>>>>>>>>>>>>>> dogtag
>>>>>>>>>>>>>> running.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We need to see more of the dogtag
debug log to see what
>>>>>>>>>>>>>> is going on.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> It looks like an authentication
problem...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL
>>>>>>>>>>>>> handshake happened
>>>>>>>>>>>>> Could not connect to LDAP server host
>>>>>>>>>>>>> seattlenfs.bpt.rocks port 636
>>>>>>>>>>>>> Error netscape.ldap.LDAPException:
Authentication failed
>>>>>>>>>>>>> (49)
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> dogtag stores its internal data in the
LDAP server and
>>>>>>>>>>>> needs to
>>>>>>>>>>>> establish a secure LDAP connection. You
can check how this
>>>>>>>>>>>> connection is configured in
/etc/pki/pki-tomcat/ca/CS.cfg,
>>>>>>>>>>>> look for
>>>>>>>>>>>> the lines:
>>>>>>>>>>>>
>>>>>>>>>>>>
internaldb.ldapauth.authtype=SslClientAuth
>>>>>>>>>>>> internaldb.ldapauth.bindDN=cn=Directory
Manager
>>>>>>>>>>>>
internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>>>>>>>
internaldb.ldapauth.clientCertNickname=subsystemCert
>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>> internaldb.ldapconn.host=vm-...
>>>>>>>>>>>> internaldb.ldapconn.port=636
>>>>>>>>>>>> internaldb.ldapconn.secureConn
>>>>>>>>>>>>
>>>>>>>>>>>> authtype can be SslClientAuth
(authentication with a ssl
>>>>>>>>>>>> certificate) or BasicAuth (authentication
with a bind DN and
>>>>>>>>>>>> password stored in
>>>>>>>>>>>>
/var/lib/pki/pki-tomcat/conf/password.conf).
>>>>>>>>>>>>
>>>>>>>>>>>> You can use this information to manually
check the
>>>>>>>>>>>> credentials. For
>>>>>>>>>>>> instance with sslclientauth:
>>>>>>>>>>>>
>>>>>>>>>>>> export
LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
>>>>>>>>>>>> export LDAPTLS_CERT='subsystemCert
cert-pki-ca'
>>>>>>>>>>>>
>>>>>>>>>>>> ldapsearch -H ldaps://`hostname`:636 -b
"" -s base -Y
>>>>>>>>>>>> EXTERNAL
>>>>>>>>>>>> (provide the password from
>>>>>>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt)
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I found this:
>>>>>>>>>>>
>>>>>>>>>>> internaldb.ldapauth.authtype=SslClientAuth
>>>>>>>>>>>
internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>>>>>>
internaldb.ldapauth.clientCertNickname=subsystemCert
>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>> internaldb.ldapconn.cloneReplicationPort=389
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>> and when I try the ldapsearch I am presented
with a prompt
>>>>>>>>>>> to provide
>>>>>>>>>>> a pin/password
>>>>>>>>>>>
>>>>>>>>>>> Please enter pin, password, or pass phrase
for security
>>>>>>>>>>> token 'ldap(0)':
>>>>>>>>>>>
>>>>>>>>>>> but there is no password file...
>>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> you are right, in 4.4. there is no pwdfile.txt
and the
>>>>>>>>>> password can be
>>>>>>>>>> found in
/var/lib/pki/pki-tomcat/conf/password.conf (with
>>>>>>>>>> the tag
>>>>>>>>>> internal=...)
>>>>>>>>>>
>>>>>>>>>> Can you check if the password with the tag
internal=...
>>>>>>>>>> allows to read
>>>>>>>>>> the keys from the NSS db?
>>>>>>>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>>> (provide password)
>>>>>>>>>
>>>>>>>>> That works...
>>>>>>>>>
>>>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias
>>>>>>>>> certutil: Checking token "NSS Certificate
DB" in slot "NSS
>>>>>>>>> User Private
>>>>>>>>> Key and Certificate Services"
>>>>>>>>> Enter Password or Pin for "NSS Certificate
DB":
>>>>>>>>> < 0> rsa
0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan)
>>>>>>>>> < 1> rsa
b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS
>>>>>>>>> Certificate
>>>>>>>>> DB:auditSigningCert cert-pki-ca
>>>>>>>>> < 2> rsa
881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert
>>>>>>>>> cert-pki-ca
>>>>>>>>> < 3> rsa
fa9a255a1d15585ac28064c0f4986e416bc48403 NSS
>>>>>>>>> Certificate
>>>>>>>>> DB:ocspSigningCert cert-pki-ca
>>>>>>>>> < 4> rsa
3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert
>>>>>>>>> cert-pki-ca
>>>>>>>>> < 5> rsa
1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS
>>>>>>>>> Certificate
>>>>>>>>> DB:subsystemCert cert-pki-ca
>>>>>>>>>
>>>>>>>>> But this doesn't (with the same password from
password.conf)
>>>>>>>>>
>>>>>>>>> # ldapsearch -H ldaps://`hostname`:636 -b
"" -s base -Y EXTERNAL
>>>>>>>>> Please enter pin, password, or pass phrase for
security token
>>>>>>>>> 'ldap(0)':
>>>>>>>>> SASL/EXTERNAL authentication started
>>>>>>>>> ldap_sasl_interactive_bind_s: Invalid credentials
(49)
>>>>>>>>>
>>>>>>>>> That password is getting me somewhere though, since
if I put
>>>>>>>>> in a
>>>>>>>>> nonsense or incorrect password it just prompts over
and over.
>>>>>>>>
>>>>>>>> Let's step back a second. You upgraded from what to
what?
>>>>>>>
>>>>>>> There wasn't much of a change... I just assumed someone
ran yum
>>>>>>> upgrade and didn't restart, then the power outage... it
looks
>>>>>>> like not much of a version change though.
>>>>>>>
>>>>>>> # grep ipa /var/log/yum.log
>>>>>>> Jan 08 04:45:32 Installed:
>>>>>>> ipa-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:45:32 Installed:
>>>>>>> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:46:06 Updated:
libipa_hbac-1.14.0-43.el7_3.4.x86_64
>>>>>>> Jan 08 04:46:07 Updated:
>>>>>>> python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
>>>>>>> Jan 08 04:46:08 Installed:
python-ipaddress-1.0.16-2.el7.noarch
>>>>>>> Jan 08 04:47:04 Installed:
>>>>>>> python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:47:05 Installed:
>>>>>>> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:47:05 Updated:
>>>>>>> ipa-admintools-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:47:05 Installed:
>>>>>>> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:47:06 Installed:
>>>>>>> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64
>>>>>>> Jan 08 04:47:17 Updated:
ipa-client-4.4.0-14.el7.centos.1.1.x86_64
>>>>>>> Jan 08 04:47:23 Updated:
ipa-server-4.4.0-14.el7.centos.1.1.x86_64
>>>>>>> Jan 08 04:47:27 Updated:
>>>>>>> ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:47:36 Installed:
>>>>>>> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch
>>>>>>> Jan 08 04:50:07 Erased:
>>>>>>> ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
>>>>>>> Jul 28 09:55:41 Updated:
ipa-common-4.4.0-14.el7.centos.7.noarch
>>>>>>> Jul 28 09:55:42 Updated:
>>>>>>> ipa-client-common-4.4.0-14.el7.centos.7.noarch
>>>>>>> Jul 28 09:55:43 Updated:
>>>>>>> ipa-server-common-4.4.0-14.el7.centos.7.noarch
>>>>>>> Jul 28 09:55:43 Updated:
>>>>>>> python2-ipalib-4.4.0-14.el7.centos.7.noarch
>>>>>>> Jul 28 09:55:44 Updated:
>>>>>>> python2-ipaclient-4.4.0-14.el7.centos.7.noarch
>>>>>>> Jul 28 09:55:45 Updated:
>>>>>>> python2-ipaserver-4.4.0-14.el7.centos.7.noarch
>>>>>>> Jul 28 09:55:45 Updated:
>>>>>>> ipa-admintools-4.4.0-14.el7.centos.7.noarch
>>>>>>> Jul 28 09:55:46 Updated:
ipa-client-4.4.0-14.el7.centos.7.x86_64
>>>>>>> Jul 28 09:55:48 Updated:
ipa-server-4.4.0-14.el7.centos.7.x86_64
>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>> ipa-server-dns-4.4.0-14.el7.centos.7.noarch
>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>> ipa-python-compat-4.4.0-14.el7.centos.7.noarch
>>>>>>>
>>>>>>> # grep pki /var/log/yum.log
>>>>>>> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch
>>>>>>> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64
>>>>>>> Jan 08 04:47:19 Installed:
pki-base-java-10.3.3-14.el7_3.noarch
>>>>>>> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64
>>>>>>> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch
>>>>>>> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch
>>>>>>> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch
>>>>>>> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch
>>>>>>> Jul 28 10:13:17 Updated:
pki-base-java-10.3.3-19.el7_3.noarch
>>>>>>> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64
>>>>>>> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch
>>>>>>> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch
>>>>>>> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Are you running in SELinux enforcing mode?
>>>>>>>
>>>>>>> # getenforce
>>>>>>> Enforcing
>>>>>>>
>>>>>>>>
>>>>>>>> If so can you run:
>>>>>>>>
>>>>>>>> # ausearch -m AVC
>>>>>>>>
>>>>>>>
>>>>>>> # ausearch -m AVC
>>>>>>> The NOLOG option to log_format is deprecated. Please use the
>>>>>>> write_logs option.
>>>>>>> The NOLOG option is overriding the write_logs current
setting.
>>>>>>> ----
>>>>>>> time->Mon Mar 14 10:39:01 2016
>>>>>>> type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e
>>>>>>> syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1
>>>>>>> a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295
>>>>>>> uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001
>>>>>>> egid=581400001 sgid=0 fsgid=581400001 tty=(none)
ses=4294967295
>>>>>>> comm="sshd" exe="/usr/sbin/sshd"
>>>>>>> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
>>>>>>> type=AVC msg=audit(1457977141.767:17537): avc: denied { read
}
>>>>>>> for pid=12965 comm="sshd"
name="authorized_keys" dev="dm-2"
>>>>>>> ino=116393517
scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
>>>>>>> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
>>>>>>>
>>>>>>>
>>>>>>>> I suspect that the subsystem cert was renewed and
everything
>>>>>>>> wasn't
>>>>>>>> updated properly. If I'm right this is unrelated to
the
>>>>>>>> upgrade it just
>>>>>>>> shone a spotlight on it.
>>>>>>>>
>>>>>>>> Can you run these commands:
>>>>>>>>
>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W
-b
>>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate
description
>>>>>>>>
>>>>>>>
>>>>>>> This returns
>>>>>>>
>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description
>>>>>>> Enter LDAP Password:
>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>> userCertificate::
>>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>> ... stuff ...
>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>>>>>>> Subsystem,O=BPT.ROCKS
>>>>>>>
>>>>>>>> and
>>>>>>>>
>>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
>>>>>>>> cert-pki-ca' | grep Serial
>>>>>>>> # certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
>>>>>>>> cert-pki-ca' -a
>>>>>>>>
>>>>>>>
>>>>>>> These both return
>>>>>>>
>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
>>>>>>> cert-pki-ca' | grep Serial
>>>>>>> Enter Password or Pin for "NSS Certificate DB":
>>>>>>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
>>>>>>> Unrecognized Object Identifier.
>>>>>>>
>>>>>>>
>>>>>>> # certutil -K -d /etc/pki/pki-tomcat/alias -n
'subsystemCert
>>>>>>> cert-pki-ca' -a
>>>>>>> certutil: Checking token "NSS Certificate DB" in
slot "NSS User
>>>>>>> Private Key and Certificate Services"
>>>>>>> Enter Password or Pin for "NSS Certificate DB":
>>>>>>> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
>>>>>>> Unrecognized Object Identifier.
>>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> you made a typo and requested the keys (-K) instead of the certs
>>>>>> (-L). Please retry with -L and you should be able to see the
>>>>>> certificate serial.
>>>>>>
>>>>>
>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory
manager'
>>>>> -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
>>>>> Enter LDAP Password:
>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>> userCertificate::
>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>> ...
>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>>>>> Subsystem,O=BPT.ROCKS
>>>>>
>>>>> So the serial is 4?
>>>>>
>>>>> [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias
>>>>> -n 'subsystemCert cert-pki-ca' | grep Serial
>>>>> Serial Number: 1341718541 (0x4ff9000d)
>>>>>
>>>>> Which is NOT 4...
>>>>>
>>>>> [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias
>>>>> -n 'subsystemCert cert-pki-ca' -a
>>>>> -----BEGIN CERTIFICATE-----
>>>>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>> ...
>>>>> sjeiFbUFtoMnfmHLIYpe5QAR
>>>>> -----END CERTIFICATE-----
>>>>>
>>>>> And this cert is NOT the same.
>>>>>
>>>> Hi,
>>>>
>>>> when certmonger renews a certificate, it runs pre-save and
>>>> post-save commands (which you can see in the output of getcert
>>>> list). In your case, the post-save command for subsystemCert
>>>> cert-pki-ca probably failed.
>>>>
>>>> This command (on the renewal master) is responsible for copying
>>>> the new cert from the NSS db to the LDAP server. You need first to
>>>> check which IPA server is the renewal master (as the serial
>>>> numbers are in a completely different range, I assume that you
>>>> have multiple IPA servers configured with a CA instance):
>>>>
>>>> $ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-`
>>>> $ kinit admin
>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b
>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN"
>>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>>>> dn:
>>>>
cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
>>>>
>>>>
>>>> In this example the renewal master is vm-ipaserver.
>>>>
>>>
>>> I get this
>>>
>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b
>>> "cn=masters,cn=ipa,cn=etc,$BASEDN"
>>> '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>>> dn:
>>> cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
>>>
>>>
>>> Which is the other FreeIPA server.
>>>
>> Hi Ian,
>>
>> it's getting difficult to follow this thread, so could we step back
>> and summarize?
> Yes, it is. Sorry.
>>
>> - On server freeipa-sea (=renewal master), the CA component is
>> installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the
>> NSSDB /etc/pki/pki-tomcat/alias
> No?
>
> [root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n
> 'subsystemCert cert-pki-ca' | grep Serial
> Serial Number: 1341718541 (0x4ff9000d)
>> - On server seattlenfs (where PKI fails to restart), the CA
>> component is installed and 'subsystemCert cert-pki-ca' has Serial
>> number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
> Yes
>
> [root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n
> 'subsystemCert cert-pki-ca' | grep Serial
> Serial Number: 1341718541 (0x4ff9000d)
>
>>
>> - In the LDAP servers, the cert is stored in the entry
>> uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the
>> cert from freeipa-sea).
> This is where the difference is. There are two certificates on the
> master and only one on the other. I wonder if this is a result of
> the slow motion replication trainwreck I have going on due to a
> zombie replica in the ldap database that no longer exists in the real
> world and I can't seem to get rid of.
>
Hi,
the remaining issue is a replication problem. There is a
"Troubleshooting replication" section [1] in IdM Admin guide that may
help you.
Regarding your zombie replica, which commands did you use to try to
remove it, and what was the output?
Flo
[1]
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/...
Hi,
To debug replication latency or problem we would need several info but
starting with the results of the following commands
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
"uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
From the output you may only copy/past the ones related to userCertificate
Also to dump the current replication status, send the result of
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
"o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
"o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
Regards
thierry
> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
manager' -W
> -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
> Enter LDAP Password:
> dn: uid=pkidbuser,ou=people,o=ipaca
> userCertificate::
> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
> <truncated>
> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
> userCertificate::
> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK
> <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR
> description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
> Subsystem
> ,O=BPT.ROCKS
>
>
> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W
> -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
> Enter LDAP Password:
> dn: uid=pkidbuser,ou=people,o=ipaca
> userCertificate::
> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
> <truncated>
> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
> Subsystem,O=BPT.RO
> CKS
>
>> If you check the certificates dates, I guess that the most recent
>> one is on server seattlenfs (because of serial numbers). Can you
>> confirm this (because our goal is to have the most recent cert
>> stored in all the NSSDBs and in LDAP)?
>>
>> Thanks,
>> Flo
>>
>>
>>>> If your renewal master is the machine on which the NSS DB
>>>> /etc/pki/pki-tomcat/alias contains the newer cert, then you can
>>>> manually run the scripts:
>>>
>>> If I'm interpreting this correctly, then it is... so I ran this on
>>> the broken machine (seattlenfs)
>>>
>>>> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad
>>>> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert
>>>> cert-pki-ca"
>>>>
>>>
>>> I ran this, it took a while, then I re-checked and the serial still
>>> seems to be 4
>>>
>>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
>>> uid=pkidbuser,ou=people,o=ipaca userCertificate description
>>> Enter LDAP Password:
>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>> userCertificate::
>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>> ...
>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>>> Subsystem,O=BPT.ROCKS
>>>
>>> Here's what was in /var/log/pki/pki-tomcat/ca/debug
>>>
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>> ============================================
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: ===== DEBUG
>>> SUBSYSTEM INITIALIZED =======
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>> ============================================
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart
>>> at autoShutdown? false
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> autoShutdown crumb file path?
>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to
>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found
>>> cert:auditSigningCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init
>>> id=debug
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> initialized debug
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> initSubsystem id=log
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to
>>> init id=log
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>> RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart
>>> at autoShutdown? false
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> autoShutdown crumb file path?
>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to
>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found
>>> cert:auditSigningCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init
>>> id=log
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> initialized log
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> initSubsystem id=jss
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to
>>> init id=jss
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart
>>> at autoShutdown? false
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> autoShutdown crumb file path?
>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to
>>> look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found
>>> cert:auditSigningCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init
>>> id=jss
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> initialized jss
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine:
>>> initSubsystem id=dbs
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to
>>> init id=dbs
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init()
>>> mEnableSerialMgmt=true
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Creating
>>> LdapBoundConnFactor(DBSubsystem)
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>> LdapBoundConnFactory: init
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>> LdapBoundConnFactory:doCloning true
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init()
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init
>>> begins
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: init: before
>>> makeConnection errorIfDown is true
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection:
>>> errorIfDown true
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>> SSLClientCertificateSelectionCB: Setting desired cert nickname to:
>>> subsystemCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket:
>>> set client auth cert nickname subsystemCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>> SSLClientCertificatSelectionCB: Entering!
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert:
>>> ocspSigningCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert:
>>> subsystemCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>> SSLClientCertificateSelectionCB: desired cert found in list:
>>> subsystemCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>> SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened
>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()
>>>
>>>
>>> I really appreciate you sticking with me through this. I owe you.
>>>
>>> - Ian
>>>
>>>> After that, check that the LDAP entry has been updated. If it is
>>>> the case, you should be able to restart pki.
>>>>
>>>> Flo
>>>>
>>>>> Thank you again for your time!
>>>>>
>>>>> Ian
>>>>>
>>>>>> Flo.
>>>>>>>
>>>>>>> which seems weird.
>>>>>>>
>>>>>>>> The description attribute in the ldapsearch output is of
the
>>>>>>>> form 2;#;DN;DN
>>>>>>>>
>>>>>>>> The # should match the serial number in the first
certutil output
>>>>>>>>
>>>>>>>> The ASCII blob (minus the BEGIN/END headers) should match
one
>>>>>>>> of the
>>>>>>>> userCertificate attributes.
>>>>>>>>
>>>>>>>> rob
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org