Hi,
I don't know if it's a copy/paste issue but the separator in your ldif file renders badly on my screen. It should be a simple dash ("-") but it looks different.

flo

On Fri, Apr 8, 2022 at 2:28 AM Yajith Dayarathna via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
Hello everyone,

I'm trying to fix an issue with our freeipa setup where multiple servers in the domain are failing to start the pki-tomcatd service.
At present we have used "pactl start --ignore-service-failure" just to get the rest of the services up and running and now trying to figure out how to fix the overall problem.

Below is a summary of the current state from what I've managed to find so far:

- We have 6 servers in our freeipa domain running version 4.5.4
- Three servers including the "IPA CA renewal master" is having an issue when starting up the pki-tomcatd service, with slightly different observations.
- Problem started at different times, other two servers starting the problem many weeks ago, which we haven't been able to fix

On the IPA CA renewal master server (this server only started having the problem few days ago following a crash > reboot ) -
- All the certificates listed in "getcert list" are valid (one closest to expiry has about 2 years left)
- Cerfificate in /etc/pki/pki-tomcat/alias/ and LDAP match along with the "description" field that has the correct serial
- Logs contain this error : Internal Database Error encountered: Could not connect to LDAP server host <FQDN> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)

On the other two servers (where the problem started weeks apart following a server reboot or a ipactl restart ) -
- Most of the certificates listed "getcert list" are already expired
- Logs contain this error : Internal Database Error encountered: Could not connect to LDAP server host <FQDN> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)
- Cerfificate in /etc/pki/pki-tomcat/alias/ and LDAP does NOT match

Other than that there are many other errors on the logs on all the servers related to replication : ERR - NSMMReplicationPlugin - send_updates"
and messages like : Certificate in file "/var/kerberos/krb5kdc/kdc.crt" is no longer valid.

To try to come up with some method to fix the servers, I've taken a clone(disconnected from the network) of a non IPA CA renewal master.
Below steps I've found on various articles were all done within the clone.

Some things I've tried within the clone out so far along with the errors:
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/

$ sudo grep internal /var/lib/pki/pki-tomcat/conf/password.conf | cut -d= -f2 > /tmp/pwdfile.txt
$ sudo certutil -K -d /etc/pki/pki-tomcat/alias -f /tmp/pwdfile.txt -n 'subsystemCert cert-pki-ca'
certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
certutil: problem listing keys: SEC_ERROR_INVALID_ARGS: security library: invalid arguments.

I did notice is that the message says "NSS Certificate DB" but on the "getcert list" this certificate shows with token='NSS FIPS 140-2 Certificate DB', not sure if there is an actual problem with the password or if I'm not using the command correctly.

Tried resetting the time back to a point where the expired certs are still valid as mentioned in https://access.redhat.com/solutions/3357261 and to get the pki-tomcatd to come up, tried to update the cert. in LDAP to match what is in /etc/pki/pki-tomcat/alias/ using this method https://access.redhat.com/solutions/3614001 fails for me.

It updates the certificate serial in "description" field but never changes the certificate blob. ldapmodify command and the contents of the .ldif file I used are below.

# ldapmodify -x -h localhost -p 389 -D “cn=directory manager” -w -f updatecert.ldif -v
ldap_initializer( ldap://localhost:389 )
replace usercertificate:
NOT ASCII (894 bytes)
replace description:
2;32;CN=Certificate Authority,O=;CN=CA Subsystem,O=<OUR.DOMAIN>
modifying entry “uid=pkidbuser,ou=people,o=ipaca”
modify complete

# cat updatecert.ldif
dn: uid=pkidbuser,ou=people,o=ipaca
changetype: modify
replace: usercertificate
usercertificate::MII..

replace: description
description: 2;32;CN=Certificate Authority,O=;CN=CA Subsystem,O=<OUR.DOMAIN>

I've used various combinations here even a delete of "userCertificate" field to add the correct one later but none of which worked for me.

I'm hoping someone can point me in the right direction.

Thanks in advance,
yajith
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure