On 8/16/17 5:24 AM, thierry bordaz via FreeIPA-users wrote:

Hi Ian

Hmm.. This looks odd.

[root@seattlenfs ianh]# ipa-csreplica-manage re-initialize --from freeipa-sea.bpt.rocks
Directory Manager password:

ipa: ERROR: Found multiple agreements for seattlenfs.bpt.rocks. Only initializing the first one returned: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\=ipaca,cn=mapping tree,cn=config
Update in progress, 15 seconds elapsed
[freeipa-sea.bpt.rocks] reports: Update failed! Status: [32  - LDAP error: No such object]

Here there are several replica agreement from freeipa-sea.bpt.rocks-> seattlenfs (for o=ipaca suffix).
I am not sure how this can happen.
It could be some remaining entries from previous install/uninstall or multiple initialization from different sources.
If it's possible to do something wrong I am sure I have done it.  I have tried the various clean-ruv commands for the apparently existing agreements with nonexistent servers many times but it seems ot have no effect.

Looking at the error below, I wonder if there were actually 2 replica agreements freeipa-sea->seattlenfs that are

Would you dump (on all servers freeipa-sea, seattlenfs and freeipa-dal):

ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nscpentrywsi


freeipa-dal no longer exists.  Also, since they also appear below... neither do edinburghnfs, fremontnis, bellevuenfs or bpt-nyc1-nfs.  The one causing me the most obvious headaches is freeipa-dal.

[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nscpentrywsi
Enter LDAP Password:
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: createTimestamp: 20160814234939Z
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: modifyTimestamp: 20170816141634Z
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freei
 pa-sea.bpt.rocks-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seat
 tlenfs.bpt.rocks-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaId: 1065
nscpentrywsi: nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsState:: KQQAAAAAAADAU5RZAAAAAAAAAAAAAAAAKgAAAAAAAAAAAAAAAAAAAA
 ==
nscpentrywsi: nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,
 cn=etc,dc=bpt,dc=rocks
nscpentrywsi: nsds5replicabinddngroupcheckinterval: 60
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: numSubordinates: 2
nscpentrywsi: nsds5ReplicaChangeCount: 961
nscpentrywsi: nsds5replicareapactive: 0

dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac
 a,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=repli
 ca,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: cn: cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat
nscpentrywsi: createTimestamp: 20160814234940Z
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: description: cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: modifyTimestamp: 20170810183437Z
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-free
 ipa-sea.bpt.rocks-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindMethod: Simple
nscpentrywsi: nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1
 NxR1NJYjNEUUVGRERBNEJDUXhNemsyTlRka05TMHhZMlE1TTJKbA0KT1MwNU1HTm1PV0V6T1MwMk5
 qazNNV0V3TndBQ0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQ0da
 bE1zY2Nod0FQeFlXMkF0Y3RhTA==}AIuTmb+o3/MWgo0mc8r8fw==
nscpentrywsi: nsDS5ReplicaHost: seattlenfs.bpt.rocks
nscpentrywsi: nsDS5ReplicaPort: 389
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaTransportInfo: TLS
nscpentrywsi: nsds50ruv: {replicageneration} 55c8f3ae000000600000
nscpentrywsi: nsds50ruv: {replica 81 ldap://seattlenfs.bpt.rocks:389} 568ac431
 000000510000 57b103ce000500510000
nscpentrywsi: nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b10
 3d4000004290000 57b254df000004290000
nscpentrywsi: nsds50ruv: {replica 1095 ldap://freeipa-sea.bpt.rocks:389} 579a9
 63c000004470000 57a42389000104470000
nscpentrywsi: nsds50ruv: {replica 96 ldap://freeipa-sea.bpt.rocks:389} 55c8f3b
 d000000600000 5799a02e000000600000
nscpentrywsi: nsds5ReplicaEnabled: on
nscpentrywsi: nsruvReplicaLastModified: {replica 81 ldap://seattlenfs.bpt.rock
 s:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.r
 ocks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1095 ldap://freeipa-sea.bpt.r
 ocks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 96 ldap://freeipa-sea.bpt.roc
 ks:389} 00000000
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsds5replicationagreement
nscpentrywsi: nsds5replicareapactive: 0
nscpentrywsi: nsds5replicaLastUpdateStart: 19700101000000Z
nscpentrywsi: nsds5replicaLastUpdateEnd: 19700101000000Z
nscpentrywsi: nsds5replicaChangesSentSinceStartup:
nscpentrywsi: nsds5replicaLastUpdateStatus: Error (32) Problem connecting to r
 eplica - LDAP error: No such object (connection error)
nscpentrywsi: nsds5replicaUpdateInProgress: FALSE
nscpentrywsi: nsds5replicaLastInitStart: 19700101000000Z
nscpentrywsi: nsds5replicaLastInitEnd: 19700101000000Z

dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac
 a,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=repli
 ca,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: cn: masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat
nscpentrywsi: createTimestamp: 20160825052011Z
nscpentrywsi: creatorsName: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: description: masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: modifyTimestamp: 20170706211220Z
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seatt
 lenfs.bpt.rocks-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindMethod: Simple
nscpentrywsi: nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1
 NxR1NJYjNEUUVGRERBNEJDUXhNemsyTlRka05TMHhZMlE1TTJKbA0KT1MwNU1HTm1PV0V6T1MwMk5
 qazNNV0V3TndBQ0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQnZ6
 Z2tGUzErSjlvdEVYRlFJaVQyMA==}yhEAKKElqudqql31/kWsUw==
nscpentrywsi: nsDS5ReplicaHost: seattlenfs.bpt.rocks
nscpentrywsi: nsDS5ReplicaPort: 389
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaTransportInfo: TLS
nscpentrywsi: nsds50ruv: {replicageneration} 55c8f3ae000000600000
nscpentrywsi: nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be80
 4c0000050a0000 587236150000050a0000
nscpentrywsi: nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b10
 3d4000004290000 57be8043000704290000
nscpentrywsi: nsds50ruv: {replica 1070 ldap://bellevuenfs.bpt.rocks:389} 57a4f
 2700000042e0000 57a4f2700007042e0000
nscpentrywsi: nsds50ruv: {replica 1075 ldap://bpt-nyc1-nfs.bpt.rocks:389} 57a4
 7865000004330000 57b111c8000404330000
nscpentrywsi: nsds50ruv: {replica 1085 ldap://fremontnis.bpt.rocks:389} 57a403
 e60000043d0000 57a643ea0000043d0000
nscpentrywsi: nsds50ruv: {replica 1090 ldap://freeipa-dal.bpt.rocks:389} 57a2d
 d35000004420000 57a2dd35000404420000
nscpentrywsi: nsds50ruv: {replica 1195 ldap://edinburghnfs.bpt.rocks:389} 57a4
 2390000004ab0000 57a42390000404ab0000
nscpentrywsi: nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.ro
 cks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.r
 ocks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1070 ldap://bellevuenfs.bpt.r
 ocks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1075 ldap://bpt-nyc1-nfs.bpt.
 rocks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1085 ldap://fremontnis.bpt.ro
 cks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1090 ldap://freeipa-dal.bpt.r
 ocks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1195 ldap://edinburghnfs.bpt.
 rocks:389} 00000000
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsds5replicationagreement
nscpentrywsi: nsds5replicareapactive: 0
nscpentrywsi: nsds5replicaLastUpdateStart: 19700101000000Z
nscpentrywsi: nsds5replicaLastUpdateEnd: 19700101000000Z
nscpentrywsi: nsds5replicaChangesSentSinceStartup:
nscpentrywsi: nsds5replicaLastUpdateStatus: Error (19) Replication error acqui
 ring replica: Replica has different database generation ID, remote replica ma
 y need to be initialized (RUV error)
nscpentrywsi: nsds5replicaUpdateInProgress: FALSE
nscpentrywsi: nsds5replicaLastInitStart: 19700101000000Z
nscpentrywsi: nsds5replicaLastInitEnd: 19700101000000Z


##################


[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" nscpentrywsi
Enter LDAP Password:
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: cn: replica
nscpentrywsi: createTimestamp: 20160825052011Z
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: modifiersName: cn=Multimaster Replication Plugin,cn=plugins,cn=c
 onfig
nscpentrywsi: modifyTimestamp: 20170810182749Z
nscpentrywsi: nsDS5Flags: 1
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seatt
 lenfs.bpt.rocks-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaId: 1290
nscpentrywsi: nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaType: 3
nscpentrywsi: nsState:: CgUAAAAAAACkpYxZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA
 ==
nscpentrywsi: nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,
 cn=etc,dc=bpt,dc=rocks
nscpentrywsi: nsds5replicabinddngroupcheckinterval: 60
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsDS5Replica
nscpentrywsi: objectClass: extensibleobject
nscpentrywsi: numSubordinates: 1
nscpentrywsi: nsds5ReplicaChangeCount: 3
nscpentrywsi: nsds5replicareapactive: 0

dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca
 ,cn=mapping tree,cn=config
nscpentrywsi: dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replic
 a,cn=o\3Dipaca,cn=mapping tree,cn=config
nscpentrywsi: cn: cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat
nscpentrywsi: createTimestamp: 20160825052011Z
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: description: cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat
nscpentrywsi: modifiersName: cn=directory manager
nscpentrywsi: modifyTimestamp: 20170810183422Z
nscpentrywsi: nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seat
 tlenfs.bpt.rocks-pki-tomcat,ou=csusers,cn=config
nscpentrywsi: nsDS5ReplicaBindMethod: Simple
nscpentrywsi: nsDS5ReplicaCredentials: {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1
 NxR1NJYjNEUUVGRERBNEJDUXhNemsyTlRka05TMHhZMlE1TTJKbA0KT1MwNU1HTm1PV0V6T1MwMk5
 qazNNV0V3TndBQ0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCQzZI
 d20zcHNMYTZKT3dQVDg4RW9XRA==}PpeZp0WYyNiA7DWqrsJfBw==
nscpentrywsi: nsDS5ReplicaHost: freeipa-sea.bpt.rocks
nscpentrywsi: nsDS5ReplicaPort: 389
nscpentrywsi: nsDS5ReplicaRoot: o=ipaca
nscpentrywsi: nsDS5ReplicaTransportInfo: TLS
nscpentrywsi: nsds50ruv: {replicageneration} 57c291d9000004290000
nscpentrywsi: nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f84
 0bf000004290000 598ca31f000104290000
nscpentrywsi: nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
nscpentrywsi: nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.r
 ocks:389} 00000000
nscpentrywsi: nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.ro
 cks:389} 00000000
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: nsds5replicationagreement
nscpentrywsi: nsds5ReplicaEnabled: on
nscpentrywsi: nsds5replicareapactive: 0
nscpentrywsi: nsds5replicaLastUpdateStart: 19700101000000Z
nscpentrywsi: nsds5replicaLastUpdateEnd: 19700101000000Z
nscpentrywsi: nsds5replicaChangesSentSinceStartup:
nscpentrywsi: nsds5replicaLastUpdateStatus: Error (19) Replication error acqui
 ring replica: Replica has different database generation ID, remote replica ma
 y need to be initialized (RUV error)
nscpentrywsi: nsds5replicaUpdateInProgress: FALSE
nscpentrywsi: nsds5replicaLastInitStart: 19700101000000Z
nscpentrywsi: nsds5replicaLastInitEnd: 19700101000000Z



best regards
thierry


On freeipa-sea and seattlenfs you may check the ipaca replica agreements status before/after reinit. I would guess they are currently reporting failures between these two replicas.

ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus

[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
Enter LDAP Password:
dn: cn=cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac
 a,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (32) Problem connecting to replica - LDAP
 error: No such object (connection error)

dn: cn=masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipac
 a,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica:
 Replica has different database generation ID, remote replica may need to be i
 nitialized (RUV error)

and

[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config" "objectClass=nsds5replicationagreement" nsds5replicaLastUpdateStatus
Enter LDAP Password:
dn: cn=cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat,cn=replica,cn=o\3Dipaca
 ,cn=mapping tree,cn=config
nsds5replicaLastUpdateStatus: Error (19) Replication error acquiring replica:
 Replica has different database generation ID, remote replica may need to be i
 nitialized (RUV error)



several times over the years when replication has stopped. 

Replication actually is working, as far as user and machine accounts and attributes are concerned anyway, and has been for a while.

There is a zombie server freeipa-dal.bpt.rocks that I can't get rid of... it shows up in the GUI on the Topology page but generates a Server not found error.  I don't know if that's related.

Zombie servers have been discussed a lot on this mailing list. You may get rid of them with list-ruv/clean-ruv subcommand. Replication usually manage to work fine even if some zombie entries exist but it is a good practice to clean them.
Yeah, I've been through every possible combination of list-ruv/clean-ruv.  I have something odd going on in that no replication agreements show up anywhere for freeipa-dal.

regards
thierry

On 08/08/2017 10:33 PM, Ian Harding via FreeIPA-users wrote:

On 8/7/17 1:44 AM, thierry bordaz wrote:



On 08/07/2017 09:22 AM, Florence Blanc-Renaud via FreeIPA-users wrote:
On 08/04/2017 11:02 PM, Ian Harding via FreeIPA-users wrote:
On 8/4/17 2:16 AM, Florence Blanc-Renaud wrote:

On 08/03/2017 11:13 PM, Ian Harding via FreeIPA-users wrote:
On 08/03/2017 12:28 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 11:51 PM, Ian Harding via FreeIPA-users wrote:
On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 01:43 AM, Ian Harding wrote:
On 08/01/2017 12:03 PM, Rob Crittenden wrote:
Ian Harding wrote:
On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 03:11 PM, Ian Harding wrote:
On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:


On 07/31/2017 11:34 AM, Rob Crittenden wrote:
Ian Harding via FreeIPA-users wrote:
I had an unexpected restart of an IPA server that had apparently had
updates run but had not been restarted. ipactl says pki-tomcatd
would
not start.

Strangely, the actual service appears to be running:


dogtag is an application within tomcat so tomcat can run without
dogtag
running.

We need to see more of the dogtag debug log to see what is going on.


It looks like an authentication problem...

[28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake happened
Could not connect to LDAP server host seattlenfs.bpt.rocks port 636
Error netscape.ldap.LDAPException: Authentication failed (49)


Hi,

dogtag stores its internal data in the LDAP server and needs to
establish a secure LDAP connection. You can check how this
connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look for
the lines:

internaldb.ldapauth.authtype=SslClientAuth
internaldb.ldapauth.bindDN=cn=Directory Manager
internaldb.ldapauth.bindPWPrompt=internaldb
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
internaldb.ldapconn.host=vm-...
internaldb.ldapconn.port=636
internaldb.ldapconn.secureConn

authtype can be SslClientAuth (authentication with a ssl
certificate) or BasicAuth (authentication with a bind DN and
password stored in /var/lib/pki/pki-tomcat/conf/password.conf).

You can use this information to manually check the credentials. For
instance with sslclientauth:

export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
export LDAPTLS_CERT='subsystemCert cert-pki-ca'

ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL
(provide the password from /etc/pki/pki-tomcat/alias/pwdfile.txt)


I found this:

internaldb.ldapauth.authtype=SslClientAuth
internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
internaldb.ldapauth.bindPWPrompt=internaldb
internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
internaldb.ldapconn.cloneReplicationPort=389
...

and when I try the ldapsearch I am presented with a prompt to provide
a pin/password

Please enter pin, password, or pass phrase for security token 'ldap(0)':

but there is no password file...

Hi,

you are right, in 4.4. there is no pwdfile.txt and the password can be
found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag
internal=...)

Can you check if the password with the tag internal=... allows to read
the keys from the NSS db?
certutil -K -d /etc/pki/pki-tomcat/alias
(provide password)

That works...

# certutil -K -d /etc/pki/pki-tomcat/alias
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private
Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
< 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64   (orphan)
< 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a   NSS Certificate
DB:auditSigningCert cert-pki-ca
< 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert
cert-pki-ca
< 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403   NSS Certificate
DB:ocspSigningCert cert-pki-ca
< 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert
cert-pki-ca
< 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856   NSS Certificate
DB:subsystemCert cert-pki-ca

But this doesn't (with the same password from password.conf)

# ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL
Please enter pin, password, or pass phrase for security token 'ldap(0)':
SASL/EXTERNAL authentication started
ldap_sasl_interactive_bind_s: Invalid credentials (49)

That password is getting me somewhere though, since if I put in a
nonsense or incorrect password it just prompts over and over.

Let's step back a second. You upgraded from what to what?

There wasn't much of a change... I just assumed someone ran yum upgrade and didn't restart, then the power outage... it looks like not much of a version change though.

# grep ipa /var/log/yum.log
Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:45:32 Installed: ipa-client-common-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64
Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch
Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:05 Installed: python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:05 Installed: ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:06 Installed: python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64
Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64
Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64
Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:47:36 Installed: ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch
Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64
Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64
Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch
Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch

# grep pki /var/log/yum.log
Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch
Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64
Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch
Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64
Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch
Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch
Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch
Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch
Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch
Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64
Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch
Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch
Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch



Are you running in SELinux enforcing mode?

# getenforce
Enforcing


If so can you run:

# ausearch -m AVC


# ausearch -m AVC
The NOLOG option to log_format is deprecated. Please use the write_logs option.
The NOLOG option is overriding the write_logs current setting.
----
time->Mon Mar 14 10:39:01 2016
type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2 success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0 items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001 suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001 tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd" subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
type=AVC msg=audit(1457977141.767:17537): avc:  denied { read } for pid=12965 comm="sshd" name="authorized_keys" dev="dm-2" ino=116393517 scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file


I suspect that the subsystem cert was renewed and everything wasn't
updated properly. If I'm right this is unrelated to the upgrade it just
shone a spotlight on it.

Can you run these commands:

$ ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate description


This returns

$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
... stuff ...
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS

and

# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' | grep Serial
# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a


These both return

# certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial
Enter Password or Pin for "NSS Certificate DB":
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.


# certutil -K -d /etc/pki/pki-tomcat/alias  -n 'subsystemCert cert-pki-ca' -a
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
Enter Password or Pin for "NSS Certificate DB":
certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID: Unrecognized Object Identifier.

Hi,

you made a typo and requested the keys (-K) instead of the certs (-L). Please retry with -L and you should be able to see the certificate serial.


[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
...
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS

So the serial is 4?

[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial
         Serial Number: 1341718541 (0x4ff9000d)

Which is NOT 4...

[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
...
sjeiFbUFtoMnfmHLIYpe5QAR
-----END CERTIFICATE-----

And this cert is NOT the same.

Hi,

when certmonger renews a certificate, it runs pre-save and post-save commands (which you can see in the output of getcert list). In your case, the post-save command for subsystemCert cert-pki-ca probably failed.

This command (on the renewal master) is responsible for copying the new cert from the NSS db to the LDAP server. You need first to check which IPA server is the renewal master (as the serial numbers are in a completely different range, I assume that you have multiple IPA servers configured with a CA instance):

$ export BASEDN=`grep basedn /etc/ipa/default.conf | cut -d= -f2-`
$ kinit admin
$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
dn: cn=CA,cn=vm-ipaserver.ipadomain.com,cn=masters,cn=ipa,cn=etc,dc=ipadomain,dc=com

In this example the renewal master is vm-ipaserver.


I get this

$ ldapsearch -LLL -Q -h `hostname` -p 389 -Y GSSAPI -b "cn=masters,cn=ipa,cn=etc,$BASEDN" '(&(cn=CA)(ipaConfigString=caRenewalMaster))' dn
dn: cn=CA,cn=freeipa-sea.bpt.rocks,cn=masters,cn=ipa,cn=etc,dc=bpt,dc=rocks

Which is the other FreeIPA server.

Hi Ian,

it's getting difficult to follow this thread, so could we step back and summarize?
Yes, it is.  Sorry.

- On server freeipa-sea (=renewal master), the CA component is installed and 'subsystemCert cert-pki-ca' has Serial number 4 in the NSSDB /etc/pki/pki-tomcat/alias
No?

[root@freeipa-sea ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial
         Serial Number: 1341718541 (0x4ff9000d)
- On server seattlenfs (where PKI fails to restart), the CA component is installed and 'subsystemCert cert-pki-ca' has Serial number 1341718541 in the NSSDB /etc/pki/pki-tomcat/alias.
Yes

[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' | grep Serial
         Serial Number: 1341718541 (0x4ff9000d)


- In the LDAP servers, the cert is stored in the entry uid=pkidbuser,ou=people,o=ipaca with Serial number 4 (i.e. it is the cert from freeipa-sea).
This is where the difference is.  There are two certificates on the master and only one on the other.  I wonder if this is a result of the slow motion replication trainwreck I have going on due to a zombie replica in the ldap database that no longer exists in the real world and I can't seem to get rid of.

Hi,

the remaining issue is a replication problem. There is a "Troubleshooting replication" section [1] in IdM Admin guide that may help you.

Regarding your zombie replica, which commands did you use to try to remove it, and what was the output?

Flo

[1] https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/trouble-gen-replication.html


Hi,

To debug replication latency or problem we would need several info but starting with the results of the following commands

[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "uid=pkidbuser,ou=people,o=ipaca" nscpentrywsi
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: entryusn;adcsn-5938e688000104290005;vucsn-5938e688000104290005:
 75274897
nscpentrywsi: modifyTimestamp;adcsn-5938e688000104290004;vucsn-5938e6880001042
 90004: 20170608055334Z
nscpentrywsi: modifiersName;adcsn-5938e688000104290003;vucsn-5938e688000104290
 003: cn=Directory Manager
nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074
nscpentrywsi: cn: pkidbuser
nscpentrywsi: createTimestamp: 20150716125146Z
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: description;vucsn-5938e688000104290001: 2;1341718541;CN=Certific
 ate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
nscpentrywsi: description;vdcsn-5938e688000104290002;deleted: 2;4;CN=Certifica
 te Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS
nscpentrywsi: mail:
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: person
nscpentrywsi: objectClass: organizationalPerson
nscpentrywsi: objectClass: inetOrgPerson
nscpentrywsi: objectClass: cmsuser
nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS
nscpentrywsi: sn: pkidbuser
nscpentrywsi: uid: pkidbuser
nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip>
 cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
nscpentrywsi: userCertificate;vucsn-5938e688000104290000:: MIIDbjCCAlagAwIBAgI
<snip>
 AR
nscpentrywsi: userPassword:: <snip>
nscpentrywsi: userstate: 1
nscpentrywsi: usertype: agentType
nscpentrywsi: parentid: 2
nscpentrywsi: entryid: 51
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi

[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca nscpentrywsi
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: dn: uid=pkidbuser,ou=people,o=ipaca
nscpentrywsi: seeAlso: CN=CA Subsystem,O=BPT.ROCKS
nscpentrywsi: userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MR
<snip>
 cBd4nOV4m9A+qRZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
nscpentrywsi: description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subs
 ystem,O=BPT.ROCKS
nscpentrywsi: entryid: 51
nscpentrywsi: parentid: 2
nscpentrywsi: modifyTimestamp: 20150716125146Z
nscpentrywsi: createTimestamp: 20150716125146Z
nscpentrywsi: modifiersName: cn=directory manager
nscpentrywsi: creatorsName: cn=directory manager
nscpentrywsi: userPassword:: <snip>
nscpentrywsi: userstate: 1
nscpentrywsi: usertype: agentType
nscpentrywsi: mail:
nscpentrywsi: cn: pkidbuser
nscpentrywsi: sn: pkidbuser
nscpentrywsi: uid: pkidbuser
nscpentrywsi: objectClass: top
nscpentrywsi: objectClass: person
nscpentrywsi: objectClass: organizationalPerson
nscpentrywsi: objectClass: inetOrgPerson
nscpentrywsi: objectClass: cmsuser
nscpentrywsi: nsUniqueId: 48809b40-2bb911e5-96a0b8b8-1fdd8074
nscpentrywsi: entryusn: 86

From the output you may only copy/past the ones related to userCertificate


Also to dump the current replication status, send the result  of
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
Enter LDAP Password:
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-freeipa-sea.bpt.roc
 ks-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaBindDN: cn=Replication Manager masterAgreement1-seattlenfs.bpt.roc
 ks-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaId: 1065
nsDS5ReplicaName: b21a1f1e-627911e6-93e6ef4b-69dcc2d1
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaType: 3
nsState:: KQQAAAAAAAAhHIpZAAAAAAEAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA==
nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt,
 dc=rocks
nsds5replicabinddngroupcheckinterval: 60
objectClass: top
objectClass: nsDS5Replica
objectClass: extensibleobject
nsds50ruv: {replicageneration} 57c291d9000004290000
nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57f840bf00000429000
 0 598a1c40000104290000
nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389}
nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-freeipa-sea.bpt.rocks-pki-tomcat;seat
 tlenfs.bpt.rocks;389;unavailable
nsds5agmtmaxcsn: o=ipaca;masterAgreement1-seattlenfs.bpt.rocks-pki-tomcat;seat
 tlenfs.bpt.rocks;389;unavailable
nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 598a
 1c16
nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000
 000
nsds5ReplicaChangeCount: 885
nsds5replicareapactive: 0
and
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"


[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b "o=ipaca" "(&(objectclass=nstombstone)(nsUniqueId=ffffffff-ffffffff-ffffffff-ffffffff))"
Enter LDAP Password:
dn: cn=replica,cn=o\3Dipaca,cn=mapping tree,cn=config
cn: replica
nsDS5Flags: 1
nsDS5ReplicaBindDN: cn=Replication Manager cloneAgreement1-seattlenfs.bpt.rock
 s-pki-tomcat,ou=csusers,cn=config
nsDS5ReplicaId: 1290
nsDS5ReplicaName: 8706ab1e-6a8311e6-a9bfdbbf-74dc7323
nsDS5ReplicaRoot: o=ipaca
nsDS5ReplicaType: 3
nsState:: CgUAAAAAAAAKFYpZAAAAAAAAAAAAAAAAKgAAAAAAAAABAAAAAAAAAA==
nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=bpt,
 dc=rocks
nsds5replicabinddngroupcheckinterval: 60
objectClass: top
objectClass: nsDS5Replica
objectClass: extensibleobject
nsds50ruv: {replicageneration} 55c8f3ae000000600000
nsds50ruv: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 57be804c0000050a0000
  587236150000050a0000
nsds50ruv: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 57b103d400000429000
 0 57be8043000704290000
nsds5agmtmaxcsn: o=ipaca;cloneAgreement1-seattlenfs.bpt.rocks-pki-tomcat;freei
 pa-sea.bpt.rocks;389;1065;587236150000050a0000
nsruvReplicaLastModified: {replica 1290 ldap://seattlenfs.bpt.rocks:389} 00000
 000
nsruvReplicaLastModified: {replica 1065 ldap://freeipa-sea.bpt.rocks:389} 0000
 0000
nsds5ReplicaChangeCount: 3
nsds5replicareapactive: 0

Regards
thierry
[root@freeipa-sea ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
  <truncated>
  ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
userCertificate:: MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQK
<truncated> fuQQZmP1XrKkrfsjeiFbUFtoMnfmHLIYpe5QAR
description: 2;1341718541;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem
  ,O=BPT.ROCKS


[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
  <truncated>
  ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.RO
  CKS

If you check the certificates dates, I guess that the most recent one is on server seattlenfs (because of serial numbers). Can you confirm this (because our goal is to have the most recent cert stored in all the NSSDBs and in LDAP)?

Thanks,
Flo


If your renewal master is the machine on which the NSS DB /etc/pki/pki-tomcat/alias contains the newer cert, then you can manually run the scripts:

If I'm interpreting this correctly, then it is... so I ran this on the broken machine (seattlenfs)

$ sudo /usr/libexec/ipa/certmonger/stop_pkicad
$ sudo /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"


I ran this, it took a while, then I re-checked and the serial still seems to be 4

$ ldapsearch -LLL -D 'cn=directory manager' -W -b uid=pkidbuser,ou=people,o=ipaca userCertificate description
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
...
  ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA Subsystem,O=BPT.ROCKS

Here's what was in /var/log/pki/pki-tomcat/ca/debug

[03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================
[03/Aug/2017:13:52:51][localhost-startStop-1]: =====  DEBUG SUBSYSTEM INITIALIZED   =======
[03/Aug/2017:13:52:51][localhost-startStop-1]: ============================================
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=debug
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized debug
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=log
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=log
[03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/signedAudit/ca_audit)
[03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/system)
[03/Aug/2017:13:52:51][localhost-startStop-1]: Creating RollingLogFile(/var/lib/pki/pki-tomcat/logs/ca/transactions)
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=log
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized log
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=jss
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=jss
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: restart at autoShutdown? false
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: autoShutdown crumb file path? /var/lib/pki/pki-tomcat/logs/autoShutdown.crumb
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: about to look for cert for auto-shutdown support:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: found cert:auditSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: done init id=jss
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initialized jss
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: initSubsystem id=dbs
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine: ready to init id=dbs
[03/Aug/2017:13:52:51][localhost-startStop-1]: DBSubsystem: init() mEnableSerialMgmt=true
[03/Aug/2017:13:52:51][localhost-startStop-1]: Creating LdapBoundConnFactor(DBSubsystem)
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory: init
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapBoundConnFactory:doCloning true
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init()
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init begins
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapAuthInfo: init ends
[03/Aug/2017:13:52:51][localhost-startStop-1]: init: before makeConnection errorIfDown is true
[03/Aug/2017:13:52:51][localhost-startStop-1]: makeConnection: errorIfDown true
[03/Aug/2017:13:52:51][localhost-startStop-1]: TCP Keep-Alive: true
[03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: LdapJssSSLSocket: set client auth cert nickname subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering!
[03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca
[03/Aug/2017:13:52:51][localhost-startStop-1]: SSL handshake happened
[03/Aug/2017:13:52:51][localhost-startStop-1]: CMSEngine.shutdown()


I really appreciate you sticking with me through this.  I owe you.

- Ian

After that, check that the LDAP entry has been updated. If it is the case, you should be able to restart pki.

Flo

Thank you again for your time!

Ian

Flo.

which seems weird.

The description attribute in the ldapsearch output is of the form 2;#;DN;DN

The # should match the serial number in the first certutil output

The ASCII blob (minus the BEGIN/END headers) should match one of the
userCertificate attributes.

rob







_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org




_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org




_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org





_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org