Hmm.. This looks odd.
[root@seattlenfs ianh]# ipa-csreplica-manage re-initialize --from
freeipa-sea.bpt.rocks
Directory Manager password:
ipa: ERROR: Found multiple agreements for seattlenfs.bpt.rocks. Only
initializing the first one returned:
cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\=ipaca,cn=mapping
tree,cn=config
Update in progress, 15 seconds elapsed
[freeipa-sea.bpt.rocks] reports: Update failed! Status: [32 - LDAP
error: No such object]
Here there are several replica agreement from freeipa-sea.bpt.rocks->
seattlenfs (for o=ipaca suffix).
I am not sure how this can happen.
It could be some remaining entries from previous install/uninstall or
multiple initialization from different sources.
Looking at the error below, I wonder if there were actually 2 replica
agreements freeipa-sea->seattlenfs that are
* cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat
* masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat
Would you dump (on all servers freeipa-sea, seattlenfs and freeipa-dal):
ldapsearch -LLL -D 'cn=directory manager' -W -b
"cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nscpentrywsi
best regards
thierry
> On freeipa-sea and seattlenfs you may check the ipaca replica
> agreements status before/after reinit. I would guess they are
> currently reporting failures between these two replicas.
>
> ldapsearch -LLL -D 'cn=directory manager' -W -b
> "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
> "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
>
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W
-b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
"objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
Enter LDAP Password:
dn:
cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac
a,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (32) Problem connecting to replica
- LDAP
error: No such object (connection error)
dn:
cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac
a,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring
replica:
Replica has different database generation ID, remote replica may need
to be i
nitialized (RUV error)
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W
-b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config"
"objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
Enter LDAP Password:
dn:
cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca
,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring
replica:
Replica has different database generation ID, remote replica may need
to be i
nitialized (RUV error)
>>
>> several times over the years when replication has stopped.
>>
>> Replication actually is working, as far as user and machine accounts
>> and attributes are concerned anyway, and has been for a while.
>>
>> There is a zombie server freeipa-dal.bpt.rocks that I can't get rid
>> of... it shows up in the GUI on the Topology page but generates a
>> Server not found error. I don't know if that's related.
>
> Zombie servers have been discussed a lot on this mailing list. You
> may get rid of them with list-ruv/clean-ruv subcommand. Replication
> usually manage to work fine even if some zombie entries exist but it
> is a good practice to clean them.
Yeah, I've been through every possible combination of
list-ruv/clean-ruv. I have something odd going on in that no
replication agreements show up anywhere for freeipa-dal.
>
> regards
> thierry
>>>
>>> On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:
>>>>
>>>> On 8/7/17 1:44 AM, thierry bordaz wrote:
>>>>
>>>>>
>>>>>
>>>>> On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users
>>>>> wrote:
>>>>>> On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
>>>>>>> On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:
>>>>>>>
>>>>>>>> On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users
wrote:
>>>>>>>>> On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
>>>>>>>>>> On 08/02/2017 11:51 PM, Ian Harding via
FreeIPA-users wrote:
>>>>>>>>>>> On 08/02/2017 12:11 AM, Florence Blanc-Renaud
wrote:
>>>>>>>>>>>> On 08/02/2017 01:43 AM, Ian Harding
wrote:
>>>>>>>>>>>>> On 08/01/2017 12:03 PM, Rob
Crittenden wrote:
>>>>>>>>>>>>>> Ian Harding wrote:
>>>>>>>>>>>>>>> On 08/01/2017 07:39 AM,
Florence Blanc-Renaud wrote:
>>>>>>>>>>>>>>>> On 08/01/2017 03:11 PM,
Ian Harding wrote:
>>>>>>>>>>>>>>>>> On 08/01/2017 01:48
AM, Florence Blanc-Renaud wrote:
>>>>>>>>>>>>>>>>>> On 08/01/2017
01:32 AM, Ian Harding via
>>>>>>>>>>>>>>>>>> FreeIPA-users
wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 07/31/2017
11:34 AM, Rob Crittenden wrote:
>>>>>>>>>>>>>>>>>>>> Ian
Harding via FreeIPA-users wrote:
>>>>>>>>>>>>>>>>>>>>> I had
an unexpected restart of an IPA server that
>>>>>>>>>>>>>>>>>>>>> had
apparently had
>>>>>>>>>>>>>>>>>>>>>
updates run but had not been restarted. ipactl
>>>>>>>>>>>>>>>>>>>>> says
pki-tomcatd
>>>>>>>>>>>>>>>>>>>>>
would
>>>>>>>>>>>>>>>>>>>>> not
start.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
Strangely, the actual service appears to be running:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> dogtag is
an application within tomcat so tomcat
>>>>>>>>>>>>>>>>>>>> can run
without
>>>>>>>>>>>>>>>>>>>> dogtag
>>>>>>>>>>>>>>>>>>>> running.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We need
to see more of the dogtag debug log to see
>>>>>>>>>>>>>>>>>>>> what is
going on.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It looks like
an authentication problem...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL
>>>>>>>>>>>>>>>>>>> handshake
happened
>>>>>>>>>>>>>>>>>>> Could not
connect to LDAP server host
>>>>>>>>>>>>>>>>>>>
seattlenfs.bpt.rocks port 636
>>>>>>>>>>>>>>>>>>> Error
netscape.ldap.LDAPException: Authentication
>>>>>>>>>>>>>>>>>>> failed (49)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> dogtag stores its
internal data in the LDAP server
>>>>>>>>>>>>>>>>>> and needs to
>>>>>>>>>>>>>>>>>> establish a
secure LDAP connection. You can check
>>>>>>>>>>>>>>>>>> how this
>>>>>>>>>>>>>>>>>> connection is
configured in
>>>>>>>>>>>>>>>>>>
/etc/pki/pki-tomcat/ca/CS.cfg, look for
>>>>>>>>>>>>>>>>>> the lines:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
internaldb.ldapauth.authtype=SslClientAuth
>>>>>>>>>>>>>>>>>>
internaldb.ldapauth.bindDN=cn=Directory Manager
>>>>>>>>>>>>>>>>>>
internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>>>>>>>>>>>>>
internaldb.ldapauth.clientCertNickname=subsystemCert
>>>>>>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>>>>>>>
internaldb.ldapconn.host=vm-...
>>>>>>>>>>>>>>>>>>
internaldb.ldapconn.port=636
>>>>>>>>>>>>>>>>>>
internaldb.ldapconn.secureConn
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> authtype can be
SslClientAuth (authentication with a
>>>>>>>>>>>>>>>>>> ssl
>>>>>>>>>>>>>>>>>> certificate) or
BasicAuth (authentication with a
>>>>>>>>>>>>>>>>>> bind DN and
>>>>>>>>>>>>>>>>>> password stored
in
>>>>>>>>>>>>>>>>>>
/var/lib/pki/pki-tomcat/conf/password.conf).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You can use this
information to manually check the
>>>>>>>>>>>>>>>>>> credentials. For
>>>>>>>>>>>>>>>>>> instance with
sslclientauth:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> export
LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
>>>>>>>>>>>>>>>>>> export
LDAPTLS_CERT='subsystemCert cert-pki-ca'
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ldapsearch -H
ldaps://`hostname`:636 -b "" -s base
>>>>>>>>>>>>>>>>>> -Y EXTERNAL
>>>>>>>>>>>>>>>>>> (provide the
password from
>>>>>>>>>>>>>>>>>>
/etc/pki/pki-tomcat/alias/pwdfile.txt)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I found this:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
internaldb.ldapauth.authtype=SslClientAuth
>>>>>>>>>>>>>>>>>
internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>>>>>>>>>>>>>
internaldb.ldapauth.clientCertNickname=subsystemCert
>>>>>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>>>>>>
internaldb.ldapconn.cloneReplicationPort=389
>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and when I try the
ldapsearch I am presented with a
>>>>>>>>>>>>>>>>> prompt to provide
>>>>>>>>>>>>>>>>> a pin/password
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Please enter pin,
password, or pass phrase for
>>>>>>>>>>>>>>>>> security token
'ldap(0)':
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> but there is no
password file...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> you are right, in 4.4.
there is no pwdfile.txt and the
>>>>>>>>>>>>>>>> password can be
>>>>>>>>>>>>>>>> found in
/var/lib/pki/pki-tomcat/conf/password.conf
>>>>>>>>>>>>>>>> (with the tag
>>>>>>>>>>>>>>>> internal=...)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you check if the
password with the tag
>>>>>>>>>>>>>>>> internal=... allows to
read
>>>>>>>>>>>>>>>> the keys from the NSS
db?
>>>>>>>>>>>>>>>> certutil -K -d
/etc/pki/pki-tomcat/alias
>>>>>>>>>>>>>>>> (provide password)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That works...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # certutil -K -d
/etc/pki/pki-tomcat/alias
>>>>>>>>>>>>>>> certutil: Checking token
"NSS Certificate DB" in slot
>>>>>>>>>>>>>>> "NSS User Private
>>>>>>>>>>>>>>> Key and Certificate
Services"
>>>>>>>>>>>>>>> Enter Password or Pin for
"NSS Certificate DB":
>>>>>>>>>>>>>>> < 0> rsa
0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan)
>>>>>>>>>>>>>>> < 1> rsa
b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS
>>>>>>>>>>>>>>> Certificate
>>>>>>>>>>>>>>> DB:auditSigningCert
cert-pki-ca
>>>>>>>>>>>>>>> < 2> rsa
881b7254c40fa40bc50681bcc8d37bb3eb49937e
>>>>>>>>>>>>>>> caSigningCert
>>>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>>>> < 3> rsa
fa9a255a1d15585ac28064c0f4986e416bc48403 NSS
>>>>>>>>>>>>>>> Certificate
>>>>>>>>>>>>>>> DB:ocspSigningCert
cert-pki-ca
>>>>>>>>>>>>>>> < 4> rsa
3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f
>>>>>>>>>>>>>>> Server-Cert
>>>>>>>>>>>>>>> cert-pki-ca
>>>>>>>>>>>>>>> < 5> rsa
1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS
>>>>>>>>>>>>>>> Certificate
>>>>>>>>>>>>>>> DB:subsystemCert cert-pki-ca
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But this doesn't (with
the same password from
>>>>>>>>>>>>>>> password.conf)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> # ldapsearch -H
ldaps://`hostname`:636 -b "" -s base -Y
>>>>>>>>>>>>>>> EXTERNAL
>>>>>>>>>>>>>>> Please enter pin, password,
or pass phrase for security
>>>>>>>>>>>>>>> token 'ldap(0)':
>>>>>>>>>>>>>>> SASL/EXTERNAL authentication
started
>>>>>>>>>>>>>>> ldap_sasl_interactive_bind_s:
Invalid credentials (49)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That password is getting me
somewhere though, since if
>>>>>>>>>>>>>>> I put in a
>>>>>>>>>>>>>>> nonsense or incorrect
password it just prompts over and
>>>>>>>>>>>>>>> over.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Let's step back a second. You
upgraded from what to what?
>>>>>>>>>>>>>
>>>>>>>>>>>>> There wasn't much of a change...
I just assumed someone
>>>>>>>>>>>>> ran yum upgrade and didn't
restart, then the power
>>>>>>>>>>>>> outage... it looks like not much of a
version change though.
>>>>>>>>>>>>>
>>>>>>>>>>>>> # grep ipa /var/log/yum.log
>>>>>>>>>>>>> Jan 08 04:45:32 Installed:
>>>>>>>>>>>>>
ipa-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:45:32 Installed:
>>>>>>>>>>>>>
ipa-client-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:46:06 Updated:
>>>>>>>>>>>>> libipa_hbac-1.14.0-43.el7_3.4.x86_64
>>>>>>>>>>>>> Jan 08 04:46:07 Updated:
>>>>>>>>>>>>>
python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
>>>>>>>>>>>>> Jan 08 04:46:08 Installed:
>>>>>>>>>>>>> python-ipaddress-1.0.16-2.el7.noarch
>>>>>>>>>>>>> Jan 08 04:47:04 Installed:
>>>>>>>>>>>>>
python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:47:05 Installed:
>>>>>>>>>>>>>
python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:47:05 Updated:
>>>>>>>>>>>>>
ipa-admintools-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:47:05 Installed:
>>>>>>>>>>>>>
ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:47:06 Installed:
>>>>>>>>>>>>>
python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:47:07 Updated:
sssd-ipa-1.14.0-43.el7_3.4.x86_64
>>>>>>>>>>>>> Jan 08 04:47:17 Updated:
>>>>>>>>>>>>>
ipa-client-4.4.0-14.el7.centos.1.1.x86_64
>>>>>>>>>>>>> Jan 08 04:47:23 Updated:
>>>>>>>>>>>>>
ipa-server-4.4.0-14.el7.centos.1.1.x86_64
>>>>>>>>>>>>> Jan 08 04:47:27 Updated:
>>>>>>>>>>>>>
ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:47:36 Installed:
>>>>>>>>>>>>>
ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch
>>>>>>>>>>>>> Jan 08 04:50:07 Erased:
>>>>>>>>>>>>>
ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
>>>>>>>>>>>>> Jul 28 09:55:41 Updated:
>>>>>>>>>>>>>
ipa-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>> Jul 28 09:55:42 Updated:
>>>>>>>>>>>>>
ipa-client-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>> Jul 28 09:55:43 Updated:
>>>>>>>>>>>>>
ipa-server-common-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>> Jul 28 09:55:43 Updated:
>>>>>>>>>>>>>
python2-ipalib-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>> Jul 28 09:55:44 Updated:
>>>>>>>>>>>>>
python2-ipaclient-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>> Jul 28 09:55:45 Updated:
>>>>>>>>>>>>>
python2-ipaserver-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>> Jul 28 09:55:45 Updated:
>>>>>>>>>>>>>
ipa-admintools-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>> Jul 28 09:55:46 Updated:
>>>>>>>>>>>>>
ipa-client-4.4.0-14.el7.centos.7.x86_64
>>>>>>>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>>>>>>>>
ipa-server-4.4.0-14.el7.centos.7.x86_64
>>>>>>>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>>>>>>>>
ipa-server-dns-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>> Jul 28 09:55:48 Updated:
>>>>>>>>>>>>>
ipa-python-compat-4.4.0-14.el7.centos.7.noarch
>>>>>>>>>>>>>
>>>>>>>>>>>>> # grep pki /var/log/yum.log
>>>>>>>>>>>>> Jan 08 04:46:05 Updated:
pki-base-10.3.3-14.el7_3.noarch
>>>>>>>>>>>>> Jan 08 04:46:09 Updated:
krb5-pkinit-1.14.1-27.el7_3.x86_64
>>>>>>>>>>>>> Jan 08 04:47:19 Installed:
>>>>>>>>>>>>> pki-base-java-10.3.3-14.el7_3.noarch
>>>>>>>>>>>>> Jan 08 04:47:19 Updated:
pki-tools-10.3.3-14.el7_3.x86_64
>>>>>>>>>>>>> Jan 08 04:47:21 Updated:
pki-server-10.3.3-14.el7_3.noarch
>>>>>>>>>>>>> Jan 08 04:47:22 Updated:
pki-kra-10.3.3-14.el7_3.noarch
>>>>>>>>>>>>> Jan 08 04:47:22 Updated:
pki-ca-10.3.3-14.el7_3.noarch
>>>>>>>>>>>>> Jul 28 10:13:16 Updated:
pki-base-10.3.3-19.el7_3.noarch
>>>>>>>>>>>>> Jul 28 10:13:17 Updated:
>>>>>>>>>>>>> pki-base-java-10.3.3-19.el7_3.noarch
>>>>>>>>>>>>> Jul 28 10:13:17 Updated:
pki-tools-10.3.3-19.el7_3.x86_64
>>>>>>>>>>>>> Jul 28 10:13:21 Updated:
pki-server-10.3.3-19.el7_3.noarch
>>>>>>>>>>>>> Jul 28 10:13:21 Updated:
pki-kra-10.3.3-19.el7_3.noarch
>>>>>>>>>>>>> Jul 28 10:13:21 Updated:
pki-ca-10.3.3-19.el7_3.noarch
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Are you running in SELinux
enforcing mode?
>>>>>>>>>>>>>
>>>>>>>>>>>>> # getenforce
>>>>>>>>>>>>> Enforcing
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If so can you run:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # ausearch -m AVC
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> # ausearch -m AVC
>>>>>>>>>>>>> The NOLOG option to log_format is
deprecated. Please use
>>>>>>>>>>>>> the write_logs option.
>>>>>>>>>>>>> The NOLOG option is overriding the
write_logs current
>>>>>>>>>>>>> setting.
>>>>>>>>>>>>> ----
>>>>>>>>>>>>> time->Mon Mar 14 10:39:01 2016
>>>>>>>>>>>>> type=SYSCALL
msg=audit(1457977141.767:17537):
>>>>>>>>>>>>> arch=c000003e syscall=2 success=no
exit=-13
>>>>>>>>>>>>> a0=7fbcab038d10 a1=800 a2=1
a3=7fbca60f42e0 items=0
>>>>>>>>>>>>> ppid=1406 pid=12965 auid=4294967295
uid=0 gid=0
>>>>>>>>>>>>> euid=581400001 suid=0 fsuid=581400001
egid=581400001
>>>>>>>>>>>>> sgid=0 fsgid=581400001 tty=(none)
ses=4294967295
>>>>>>>>>>>>> comm="sshd"
exe="/usr/sbin/sshd"
>>>>>>>>>>>>>
subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
>>>>>>>>>>>>> type=AVC
msg=audit(1457977141.767:17537): avc: denied {
>>>>>>>>>>>>> read } for pid=12965
comm="sshd" name="authorized_keys"
>>>>>>>>>>>>> dev="dm-2" ino=116393517
>>>>>>>>>>>>>
scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
>>>>>>>>>>>>>
tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I suspect that the subsystem cert
was renewed and
>>>>>>>>>>>>>> everything wasn't
>>>>>>>>>>>>>> updated properly. If I'm
right this is unrelated to the
>>>>>>>>>>>>>> upgrade it just
>>>>>>>>>>>>>> shone a spotlight on it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you run these commands:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> $ ldapsearch -LLL -D
'cn=directory manager' -W -b
>>>>>>>>>>>>>> uid=pkidbuser,ou=people,o=ipaca
userCertificate description
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This returns
>>>>>>>>>>>>>
>>>>>>>>>>>>> $ ldapsearch -LLL -D
'cn=directory manager' -W -b
>>>>>>>>>>>>> uid=pkidbuser,ou=people,o=ipaca
userCertificate description
>>>>>>>>>>>>> Enter LDAP Password:
>>>>>>>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>>>>> userCertificate::
>>>>>>>>>>>>>
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>>>>>>> ... stuff ...
>>>>>>>>>>>>>
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>>>>>>>> description: 2;4;CN=Certificate
>>>>>>>>>>>>> Authority,O=BPT.ROCKS;CN=CA
Subsystem,O=BPT.ROCKS
>>>>>>>>>>>>>
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> # certutil -L -d
/etc/pki/pki-tomcat/alias -n
>>>>>>>>>>>>>> 'subsystemCert
>>>>>>>>>>>>>> cert-pki-ca' | grep Serial
>>>>>>>>>>>>>> # certutil -L -d
/etc/pki/pki-tomcat/alias -n
>>>>>>>>>>>>>> 'subsystemCert
cert-pki-ca' -a
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> These both return
>>>>>>>>>>>>>
>>>>>>>>>>>>> # certutil -K -d
/etc/pki/pki-tomcat/alias -n
>>>>>>>>>>>>> 'subsystemCert cert-pki-ca' |
grep Serial
>>>>>>>>>>>>> Enter Password or Pin for "NSS
Certificate DB":
>>>>>>>>>>>>> certutil: problem listing keys:
>>>>>>>>>>>>> SEC_ERROR_UNRECOGNIZED_OID:
Unrecognized Object Identifier.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> # certutil -K -d
/etc/pki/pki-tomcat/alias -n
>>>>>>>>>>>>> 'subsystemCert cert-pki-ca'
-a
>>>>>>>>>>>>> certutil: Checking token "NSS
Certificate DB" in slot
>>>>>>>>>>>>> "NSS User Private Key and
Certificate Services"
>>>>>>>>>>>>> Enter Password or Pin for "NSS
Certificate DB":
>>>>>>>>>>>>> certutil: problem listing keys:
>>>>>>>>>>>>> SEC_ERROR_UNRECOGNIZED_OID:
Unrecognized Object Identifier.
>>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> you made a typo and requested the keys
(-K) instead of the
>>>>>>>>>>>> certs (-L). Please retry with -L and you
should be able to
>>>>>>>>>>>> see the certificate serial.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D
'cn=directory
>>>>>>>>>>> manager' -W -b
uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>>> userCertificate description
>>>>>>>>>>> Enter LDAP Password:
>>>>>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>>>> userCertificate::
>>>>>>>>>>>
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>>>>> ...
>>>>>>>>>>>
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>>>>>> description: 2;4;CN=Certificate
Authority,O=BPT.ROCKS;CN=CA
>>>>>>>>>>> Subsystem,O=BPT.ROCKS
>>>>>>>>>>>
>>>>>>>>>>> So the serial is 4?
>>>>>>>>>>>
>>>>>>>>>>> [root@seattlenfs ianh]# certutil -L -d
>>>>>>>>>>> /etc/pki/pki-tomcat/alias -n
'subsystemCert cert-pki-ca' |
>>>>>>>>>>> grep Serial
>>>>>>>>>>> Serial Number: 1341718541
(0x4ff9000d)
>>>>>>>>>>>
>>>>>>>>>>> Which is NOT 4...
>>>>>>>>>>>
>>>>>>>>>>> [root@seattlenfs ianh]# certutil -L -d
>>>>>>>>>>> /etc/pki/pki-tomcat/alias -n
'subsystemCert cert-pki-ca' -a
>>>>>>>>>>> -----BEGIN CERTIFICATE-----
>>>>>>>>>>>
MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>> sjeiFbUFtoMnfmHLIYpe5QAR
>>>>>>>>>>> -----END CERTIFICATE-----
>>>>>>>>>>>
>>>>>>>>>>> And this cert is NOT the same.
>>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> when certmonger renews a certificate, it runs
pre-save and
>>>>>>>>>> post-save commands (which you can see in the
output of
>>>>>>>>>> getcert list). In your case, the post-save
command for
>>>>>>>>>> subsystemCert cert-pki-ca probably failed.
>>>>>>>>>>
>>>>>>>>>> This command (on the renewal master) is
responsible for
>>>>>>>>>> copying the new cert from the NSS db to the LDAP
server. You
>>>>>>>>>> need first to check which IPA server is the
renewal master
>>>>>>>>>> (as the serial numbers are in a completely
different range,
>>>>>>>>>> I assume that you have multiple IPA servers
configured with
>>>>>>>>>> a CA instance):
>>>>>>>>>>
>>>>>>>>>> $ export BASEDN=`grep basedn
/etc/ipa/default.conf | cut -d=
>>>>>>>>>> -f2-`
>>>>>>>>>> $ kinit admin
>>>>>>>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y
GSSAPI -b
>>>>>>>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN"
>>>>>>>>>>
'(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>>>>>>>>>> dn:
>>>>>>>>>>
cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In this example the renewal master is
vm-ipaserver.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I get this
>>>>>>>>>
>>>>>>>>> $ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI
-b
>>>>>>>>> "cn=masters,cn=ipa,cn=etc,$BASEDN"
>>>>>>>>>
'(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
>>>>>>>>> dn:
>>>>>>>>>
cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Which is the other FreeIPA server.
>>>>>>>>>
>>>>>>>> Hi Ian,
>>>>>>>>
>>>>>>>> it's getting difficult to follow this thread, so
could we step
>>>>>>>> back and summarize?
>>>>>>> Yes, it is. Sorry.
>>>>>>>>
>>>>>>>> - On server freeipa-sea (=renewal master), the CA
component is
>>>>>>>> installed and 'subsystemCert cert-pki-ca' has
Serial number 4
>>>>>>>> in the NSSDB /etc/pki/pki-tomcat/alias
>>>>>>> No?
>>>>>>>
>>>>>>> [root@freeipa-sea ianh]# certutil -L -d
>>>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' | grep
>>>>>>> Serial
>>>>>>> Serial Number: 1341718541 (0x4ff9000d)
>>>>>>>> - On server seattlenfs (where PKI fails to restart), the
CA
>>>>>>>> component is installed and 'subsystemCert
cert-pki-ca' has
>>>>>>>> Serial number 1341718541 in the NSSDB
/etc/pki/pki-tomcat/alias.
>>>>>>> Yes
>>>>>>>
>>>>>>> [root@seattlenfs ianh]# certutil -L -d
>>>>>>> /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' | grep
>>>>>>> Serial
>>>>>>> Serial Number: 1341718541 (0x4ff9000d)
>>>>>>>
>>>>>>>>
>>>>>>>> - In the LDAP servers, the cert is stored in the entry
>>>>>>>> uid=pkidbuser,ou=people,o=ipaca with Serial number 4
(i.e. it
>>>>>>>> is the cert from freeipa-sea).
>>>>>>> This is where the difference is. There are two certificates
on
>>>>>>> the master and only one on the other. I wonder if this is a
>>>>>>> result of the slow motion replication trainwreck I have going
>>>>>>> on due to a zombie replica in the ldap database that no
longer
>>>>>>> exists in the real world and I can't seem to get rid of.
>>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> the remaining issue is a replication problem. There is a
>>>>>> "Troubleshooting replication" section [1] in IdM Admin
guide
>>>>>> that may help you.
>>>>>>
>>>>>> Regarding your zombie replica, which commands did you use to try
>>>>>> to remove it, and what was the output?
>>>>>>
>>>>>> Flo
>>>>>>
>>>>>> [1]
>>>>>>
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/...
>>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> To debug replication latency or problem we would need several
>>>>> info but starting with the results of the following commands
>>>>>
>>>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
>>>>> manager' -W -b "uid=pkidbuser,ou=people,o=ipaca"
nscpentrywsi
>>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
manager'
>>>> -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
>>>> Enter LDAP Password:
>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>> nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca
>>>> nscpentrywsi:
>>>> entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005:
>>>> 75274897
>>>> nscpentrywsi:
>>>> modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042
>>>> 90004: 20170608055334Z
>>>> nscpentrywsi:
>>>> modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290
>>>> 003: cn=Directory Manager
>>>> nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074
>>>> nscpentrywsi: cn: pkidbuser
>>>> nscpentrywsi: createTimestamp: 20150716125146Z
>>>> nscpentrywsi: creatorsName: cn=directory manager
>>>> nscpentrywsi: description;vucsn-5938e688000104290001:
>>>> 2;1341718541;CN=Certific
>>>> ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
>>>> nscpentrywsi: description;vdcsn-5938e688000104290002;deleted:
>>>> 2;4;CN=Certifica
>>>> te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
>>>> nscpentrywsi: mail:
>>>> nscpentrywsi: objectClass: top
>>>> nscpentrywsi: objectClass: person
>>>> nscpentrywsi: objectClass: organizationalPerson
>>>> nscpentrywsi: objectClass: inetOrgPerson
>>>> nscpentrywsi: objectClass: cmsuser
>>>> nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS
>>>> nscpentrywsi: sn: pkidbuser
>>>> nscpentrywsi: uid: pkidbuser
>>>> nscpentrywsi: userCertificate::
>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
>>>> <snip>
>>>> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>> nscpentrywsi: userCertificate;vucsn-5938e688000104290000::
>>>> MIIDbjCCAlagAwIBAgI
>>>> <snip>
>>>> AR
>>>> nscpentrywsi: userPassword:: <snip>
>>>> nscpentrywsi: userstate: 1
>>>> nscpentrywsi: usertype: agentType
>>>> nscpentrywsi: parentid: 2
>>>> nscpentrywsi: entryid: 51
>>>>> and
>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory
manager'
>>>>> -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
>>>>
>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager'
>>>> -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
>>>> Enter LDAP Password:
>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>> nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca
>>>> nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS
>>>> nscpentrywsi: userCertificate::
>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
>>>> <snip>
>>>> cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>> nscpentrywsi: description: 2;4;CN=Certificate
>>>> Authority,O=BPT.ROCKS;CN=CA Subs
>>>> ystem,O=BPT.ROCKS
>>>> nscpentrywsi: entryid: 51
>>>> nscpentrywsi: parentid: 2
>>>> nscpentrywsi: modifyTimestamp: 20150716125146Z
>>>> nscpentrywsi: createTimestamp: 20150716125146Z
>>>> nscpentrywsi: modifiersName: cn=directory manager
>>>> nscpentrywsi: creatorsName: cn=directory manager
>>>> nscpentrywsi: userPassword:: <snip>
>>>> nscpentrywsi: userstate: 1
>>>> nscpentrywsi: usertype: agentType
>>>> nscpentrywsi: mail:
>>>> nscpentrywsi: cn: pkidbuser
>>>> nscpentrywsi: sn: pkidbuser
>>>> nscpentrywsi: uid: pkidbuser
>>>> nscpentrywsi: objectClass: top
>>>> nscpentrywsi: objectClass: person
>>>> nscpentrywsi: objectClass: organizationalPerson
>>>> nscpentrywsi: objectClass: inetOrgPerson
>>>> nscpentrywsi: objectClass: cmsuser
>>>> nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074
>>>> nscpentrywsi: entryusn: 86
>>>>>
>>>>> From the output you may only copy/past the ones related to
>>>>> userCertificate
>>>>>
>>>>>
>>>>> Also to dump the current replication status, send the result of
>>>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
>>>>> manager' -W -b
>>>>>
"o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
manager'
>>>> -W -b "o=ipaca"
>>>>
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>>> Enter LDAP Password:
>>>> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>>>> cn: replica
>>>> nsDS5Flags: 1
>>>> nsDS5ReplicaBindDN: cn=Replication Manager
>>>> cloneAgreement1-freeipa-sea.bpt.roc
>>>> ks-pki-tomcat,ou=csusers,cn=config
>>>> nsDS5ReplicaBindDN: cn=Replication Manager
>>>> masterAgreement1-seattlenfs.bpt.roc
>>>> ks-pki-tomcat,ou=csusers,cn=config
>>>> nsDS5ReplicaId: 1065
>>>> nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1
>>>> nsDS5ReplicaRoot: o=ipaca
>>>> nsDS5ReplicaType: 3
>>>> nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA==
>>>> nsds5replicabinddngroup: cn=replication
>>>> managers,cn=sysaccounts,cn=etc,dc=bpt,
>>>> dc=rocks
>>>> nsds5replicabinddngroupcheckinterval: 60
>>>> objectClass: top
>>>> objectClass: nsDS5Replica
>>>> objectClass: extensibleobject
>>>> nsds50ruv: {replicageneration} 57c291d9000004290000
>>>> nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
>>>> 57f840bf00000429000
>>>> 0 598a1c40000104290000
>>>> nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
>>>> nsds5agmtmaxcsn:
>>>> o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat
>>>> tlenfs.bpt.rocks;389;unavailable
>>>> nsds5agmtmaxcsn:
>>>> o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat
>>>> tlenfs.bpt.rocks;389;unavailable
>>>> nsruvReplicaLastModified: {replica 1065
>>>> ldap://freeipa-sea.bpt.rocks:389} 598a
>>>> 1c16
>>>> nsruvReplicaLastModified: {replica 1290
>>>> ldap://seattlenfs.bpt.rocks:389} 00000
>>>> 000
>>>> nsds5ReplicaChangeCount: 885
>>>> nsds5replicareapactive: 0
>>>>> and
>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory
manager'
>>>>> -W -b
>>>>>
"o=ipaca"//"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>>>
>>>>
>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager'
>>>> -W -b "o=ipaca"
>>>>
"(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
>>>> Enter LDAP Password:
>>>> dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
>>>> cn: replica
>>>> nsDS5Flags: 1
>>>> nsDS5ReplicaBindDN: cn=Replication Manager
>>>> cloneAgreement1-seattlenfs.bpt.rock
>>>> s-pki-tomcat,ou=csusers,cn=config
>>>> nsDS5ReplicaId: 1290
>>>> nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323
>>>> nsDS5ReplicaRoot: o=ipaca
>>>> nsDS5ReplicaType: 3
>>>> nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA==
>>>> nsds5replicabinddngroup: cn=replication
>>>> managers,cn=sysaccounts,cn=etc,dc=bpt,
>>>> dc=rocks
>>>> nsds5replicabinddngroupcheckinterval: 60
>>>> objectClass: top
>>>> objectClass: nsDS5Replica
>>>> objectClass: extensibleobject
>>>> nsds50ruv: {replicageneration} 55c8f3ae000000600000
>>>> nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
>>>> 57be804c0000050a0000
>>>> 587236150000050a0000
>>>> nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389}
>>>> 57b103d400000429000
>>>> 0 57be8043000704290000
>>>> nsds5agmtmaxcsn:
>>>> o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei
>>>> pa-sea.bpt.rocks;389;1065;587236150000050a0000
>>>> nsruvReplicaLastModified: {replica 1290
>>>> ldap://seattlenfs.bpt.rocks:389} 00000
>>>> 000
>>>> nsruvReplicaLastModified: {replica 1065
>>>> ldap://freeipa-sea.bpt.rocks:389} 0000
>>>> 0000
>>>> nsds5ReplicaChangeCount: 3
>>>> nsds5replicareapactive: 0
>>>>>
>>>>> Regards
>>>>> thierry
>>>>>>> [root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory
>>>>>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca
userCertificate
>>>>>>> description
>>>>>>> Enter LDAP Password:
>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>> userCertificate::
>>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>> <truncated>
>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>> userCertificate::
>>>>>>> MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK
>>>>>>> <truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR
>>>>>>> description: 2;1341718541;CN=Certificate
>>>>>>> Authority,O=BPT.ROCKS;CN=CA Subsystem
>>>>>>> ,O=BPT.ROCKS
>>>>>>>
>>>>>>>
>>>>>>> [root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory
>>>>>>> manager' -W -b uid=pkidbuser,ou=people,o=ipaca
userCertificate
>>>>>>> description
>>>>>>> Enter LDAP Password:
>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>> userCertificate::
>>>>>>> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>> <truncated>
>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
>>>>>>> Subsystem,O=BPT.RO
>>>>>>> CKS
>>>>>>>
>>>>>>>> If you check the certificates dates, I guess that the
most
>>>>>>>> recent one is on server seattlenfs (because of serial
>>>>>>>> numbers). Can you confirm this (because our goal is to
have
>>>>>>>> the most recent cert stored in all the NSSDBs and in
LDAP)?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Flo
>>>>>>>>
>>>>>>>>
>>>>>>>>>> If your renewal master is the machine on which
the NSS DB
>>>>>>>>>> /etc/pki/pki-tomcat/alias contains the newer
cert, then you
>>>>>>>>>> can manually run the scripts:
>>>>>>>>>
>>>>>>>>> If I'm interpreting this correctly, then it is...
so I ran
>>>>>>>>> this on the broken machine (seattlenfs)
>>>>>>>>>
>>>>>>>>>> $ sudo /usr/libexec/ipa/certmonger/stop_pkicad
>>>>>>>>>> $ sudo /usr/libexec/ipa/certmonger/renew_ca_cert
>>>>>>>>>> "subsystemCert cert-pki-ca"
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I ran this, it took a while, then I re-checked and
the serial
>>>>>>>>> still seems to be 4
>>>>>>>>>
>>>>>>>>> $ ldapsearch -LLL -D 'cn=directory manager'
-W -b
>>>>>>>>> uid=pkidbuser,ou=people,o=ipaca userCertificate
description
>>>>>>>>> Enter LDAP Password:
>>>>>>>>> dn: uid=pkidbuser,ou=people,o=ipaca
>>>>>>>>> userCertificate::
>>>>>>>>>
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
>>>>>>>>> ...
>>>>>>>>> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
>>>>>>>>> description: 2;4;CN=Certificate
Authority,O=BPT.ROCKS;CN=CA
>>>>>>>>> Subsystem,O=BPT.ROCKS
>>>>>>>>>
>>>>>>>>> Here's what was in
/var/log/pki/pki-tomcat/ca/debug
>>>>>>>>>
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> ============================================
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: =====
DEBUG
>>>>>>>>> SUBSYSTEM INITIALIZED =======
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> ============================================
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> restart at autoShutdown? false
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> autoShutdown crumb file path?
>>>>>>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> about to look for cert for auto-shutdown
>>>>>>>>> support:auditSigningCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> found cert:auditSigningCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> done init id=debug
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> initialized debug
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> initSubsystem id=log
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> ready to init id=log
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
Creating
>>>>>>>>>
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
>>>>>>>>>
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
Creating
>>>>>>>>>
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
Creating
>>>>>>>>>
RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> restart at autoShutdown? false
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> autoShutdown crumb file path?
>>>>>>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> about to look for cert for auto-shutdown
>>>>>>>>> support:auditSigningCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> found cert:auditSigningCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> done init id=log
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> initialized log
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> initSubsystem id=jss
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> ready to init id=jss
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> restart at autoShutdown? false
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> autoShutdown crumb file path?
>>>>>>>>> /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> about to look for cert for auto-shutdown
>>>>>>>>> support:auditSigningCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> found cert:auditSigningCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> done init id=jss
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> initialized jss
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> initSubsystem id=dbs
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
CMSEngine:
>>>>>>>>> ready to init id=dbs
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
DBSubsystem:
>>>>>>>>> init() mEnableSerialMgmt=true
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
Creating
>>>>>>>>> LdapBoundConnFactor(DBSubsystem)
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> LdapBoundConnFactory: init
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> LdapBoundConnFactory:doCloning true
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
LdapAuthInfo:
>>>>>>>>> init()
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
LdapAuthInfo:
>>>>>>>>> init begins
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
LdapAuthInfo:
>>>>>>>>> init ends
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: init:
before
>>>>>>>>> makeConnection errorIfDown is true
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> makeConnection: errorIfDown true
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: TCP
>>>>>>>>> Keep-Alive: true
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> SSLClientCertificateSelectionCB: Setting desired cert
>>>>>>>>> nickname to: subsystemCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> LdapJssSSLSocket: set client auth cert nickname
subsystemCert
>>>>>>>>> cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> SSLClientCertificatSelectionCB: Entering!
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
Candidate
>>>>>>>>> cert: ocspSigningCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
Candidate
>>>>>>>>> cert: subsystemCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> SSLClientCertificateSelectionCB: desired cert found
in list:
>>>>>>>>> subsystemCert cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> SSLClientCertificateSelectionCB: returning:
subsystemCert
>>>>>>>>> cert-pki-ca
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]: SSL
handshake
>>>>>>>>> happened
>>>>>>>>> [03/Aug/2017:13:52:51][localhost-startStop-1]:
>>>>>>>>> CMSEngine.shutdown()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I really appreciate you sticking with me through
this. I owe
>>>>>>>>> you.
>>>>>>>>>
>>>>>>>>> - Ian
>>>>>>>>>
>>>>>>>>>> After that, check that the LDAP entry has been
updated. If
>>>>>>>>>> it is the case, you should be able to restart
pki.
>>>>>>>>>>
>>>>>>>>>> Flo
>>>>>>>>>>
>>>>>>>>>>> Thank you again for your time!
>>>>>>>>>>>
>>>>>>>>>>> Ian
>>>>>>>>>>>
>>>>>>>>>>>> Flo.
>>>>>>>>>>>>>
>>>>>>>>>>>>> which seems weird.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The description attribute in the
ldapsearch output is of
>>>>>>>>>>>>>> the form 2;#;DN;DN
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The # should match the serial
number in the first
>>>>>>>>>>>>>> certutil output
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The ASCII blob (minus the
BEGIN/END headers) should
>>>>>>>>>>>>>> match one of the
>>>>>>>>>>>>>> userCertificate attributes.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> rob
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>>> To unsubscribe send an email to
>>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>>> _______________________________________________
>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>> To unsubscribe send an email to
>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> FreeIPA-users mailing list --freeipa-users(a)lists.fedorahosted.org
>>>> To unsubscribe send an email
tofreeipa-users-leave(a)lists.fedorahosted.org
>>>
>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list --freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email tofreeipa-users-leave(a)lists.fedorahosted.org
>