Hello everyone,
I'm trying to fix an issue with our freeipa setup where multiple servers in the domain
are failing to start the pki-tomcatd service.
At present we have used "pactl start --ignore-service-failure" just to get the
rest of the services up and running and now trying to figure out how to fix the overall
problem.
Below is a summary of the current state from what I've managed to find so far:
- We have 6 servers in our freeipa domain running version 4.5.4
- Three servers including the "IPA CA renewal master" is having an issue when
starting up the pki-tomcatd service, with slightly different observations.
- Problem started at different times, other two servers starting the problem many weeks
ago, which we haven't been able to fix
On the IPA CA renewal master server (this server only started having the problem few days
ago following a crash > reboot ) -
- All the certificates listed in "getcert list" are valid (one closest to expiry
has about 2 years left)
- Cerfificate in /etc/pki/pki-tomcat/alias/ and LDAP match along with the
"description" field that has the correct serial
- Logs contain this error : Internal Database Error encountered: Could not connect to LDAP
server host <FQDN> port 636 Error netscape.ldap.LDAPException: Authentication failed
(49)
On the other two servers (where the problem started weeks apart following a server reboot
or a ipactl restart ) -
- Most of the certificates listed "getcert list" are already expired
- Logs contain this error : Internal Database Error encountered: Could not connect to LDAP
server host <FQDN> port 636 Error netscape.ldap.LDAPException: Authentication failed
(49)
- Cerfificate in /etc/pki/pki-tomcat/alias/ and LDAP does NOT match
Other than that there are many other errors on the logs on all the servers related to
replication : ERR - NSMMReplicationPlugin - send_updates"
and messages like : Certificate in file "/var/kerberos/krb5kdc/kdc.crt" is no
longer valid.
To try to come up with some method to fix the servers, I've taken a clone(disconnected
from the network) of a non IPA CA renewal master.
Below steps I've found on various articles were all done within the clone.
Some things I've tried within the clone out so far along with the errors:
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
$ sudo grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 >
/tmp/pwdfile.txt
$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert
cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key
and Certificate Services"
certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid
arguments.
I did notice is that the message says "NSS Certificate DB" but on the
"getcert list" this certificate shows with token='NSS FIPS 140-2 Certificate
DB', not sure if there is an actual problem with the password or if I'm not using
the command correctly.
Tried resetting the time back to a point where the expired certs are still valid as
mentioned in
https://access.redhat.com/solutions/3357261 and to get the pki-tomcatd to
come up, tried to update the cert. in LDAP to match what is in /etc/pki/pki-tomcat/alias/
using this method
https://access.redhat.com/solutions/3614001 fails for me.
It updates the certificate serial in "description" field but never changes the
certificate blob. ldapmodify command and the contents of the .ldif file I used are below.
# ldapmodify -x -h localhost -p 389 -D “cn=directory manager” -w -f updatecert.ldif -v
ldap_initializer( ldap://localhost:389 )
replace usercertificate:
NOT ASCII (894 bytes)
replace description:
2;32;CN=Certificate Authority,O=;CN=CA Subsystem,O=<OUR.DOMAIN>
modifying entry “uid=pkidbuser,ou=people,o=ipaca”
modify complete
# cat updatecert.ldif
dn: uid=pkidbuser,ou=people,o=ipaca
changetype: modify
replace: usercertificate
usercertificate::MII..
–
replace: description
description: 2;32;CN=Certificate Authority,O=;CN=CA Subsystem,O=<OUR.DOMAIN>
I've used various combinations here even a delete of "userCertificate" field
to add the correct one later but none of which worked for me.
I'm hoping someone can point me in the right direction.
Thanks in advance,
yajith