Dear Rob,
Thanks for your pointers.
I did a
grep -v INFO debug.*
And found out that although there have been many "Connection refused"
errors for a long time (which could be due to a different issue - the
system worked so far), only in the past few days the error has become
"Authentication failed (48)".
In the meantime I was also browsing
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
which helped me find that the certificate that tomcat uses to authenticate
to LDAP has expired on 11-11-2022:
[root@ipa02 /]# certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca'
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1610547202 (0x5fff0002)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: "CN=Certificate Authority,O=HQ.SPINQUE.COM"
Validity:
Not Before: Sat Nov 21 12:56:43 2020
Not After : Fri Nov 11 12:56:43 2022
I'm not sure why it was not renewed, but now that it is in this state, what
would be the correct procedure to renew it?
Best, Roberto
On Tue, 15 Nov 2022 at 19:47, Rob Crittenden <rcritten(a)redhat.com> wrote:
Roberto Cornacchia via FreeIPA-users wrote:
> Hi there,
>
> I appear to be stuck in a failing upgrade.
>
> On Rocky Linux 8.6. The server is one of 2 replicas, both CA and DNS
> servers.
>
> It all started with pki-tomcat being down on a running server
> (
ipa02.hq.spinque.com <
http://ipa02.hq.spinque.com>):
>
> ipactl status
> Directory Service: RUNNING
> krb5kdc Service: RUNNING
> kadmin Service: RUNNING
> named Service: RUNNING
> httpd Service: RUNNING
> ipa-custodia Service: RUNNING
> pki-tomcatd Service: STOPPED
> ipa-otpd Service: RUNNING
> ipa-dnskeysyncd Service: RUNNING
> 1 service(s) are not running
>
> and unable to go up again, with these errors:
>
> ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error:
> for url:
http://ipa02.hq.spinque.com:8080/ca/admin/ca/getStatus
>
> SEVERE: LdapBoundConnFactory: Unable to connect to LDAP server:
> Authentication failed
> netscape.ldap.LDAPException: Authentication failed (48)
>
> Having read something about a similar issue being caused by nss 3.67
> (the one installed in the system), I ran a dnf update (4.9.8-8
installed).
>
> This actually complicated things, because now it still fails, but also
> it tries to upgrade every time it starts, failing the upgrade. As far as
> I can see in the upgrade log, The actual upgrade succeeds, but starting
> the services at the end fails, which makes the whole procedure fail.
>
> So running ipactl restart --ignore-service-failures does not help,
> because the automatic upgrade fails and that stops all the services as a
> last step.
>
> I'm not sure how I could continue, some pointer would be appreciated.
>
> Errors I see now:
>
> ERR - set_krb5_creds - Could not get initial credentials for principal
> [ldap/ipa02.hq.spinque.com(a)HQ.SPINQUE.COM
> <mailto:ipa02.hq.spinque.com@HQ.SPINQUE.COM>] in keytab
> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
> requested realm)
>
> ldap_child[2130]: Failed to initialize credentials using keytab
> [MEMORY:/etc/krb5.keytab]: Cannot contact any KDC for realm
> 'HQ.SPINQUE.COM <
http://HQ.SPINQUE.COM>';. Unable to create
> GSSAPI-encrypted LDAP connection.
Add --skip-version-check to not force an upgrade.
You need to determine why the CA won't start. See the journal and/or
/var/log/pki/pki-tomcat/ca/debug*.
The trick with the CA debug log is to start looking where the last
server start is and move downwards in the file. Starting at the tail
usually isn't fruitful.
rob