When I upgraded the servers to EL8 (I rebuilt from scratch using the old hostnames), I had neglected to assign an IPA CA renewal master after the old “boss” was retired. This crime is of course it’s own punishment.
I found the documentation for handling this to actually be pretty good. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
fraser’s blog was also helpful (in confirming I executed this correctly) https://frasertweedale.github.io/blog-redhat/posts/2019-05-24-ipa-cert-fix.h...
I progressed through the other three IPA servers, but the last one still has a bad expiration on the CA cert.
[root@ef-idm01 ~]# date Wed Feb 14 07:08:38 PST 2024 [root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir' Request ID '20230530175932': status: MONITORING subject: CN=ef-idm01.production.efilm.comhttp://ef-idm01.production.efilm.com,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2025-05-30 10:59:53 PDT Request ID '20230530180022': status: MONITORING subject: CN=ef-idm01.production.efilm.comhttp://ef-idm01.production.efilm.com,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2025-05-30 11:00:30 PDT Request ID '20230530180438': status: NEED_CA subject: CN=IPA RA,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2024-01-02 07:58:28 PST [root@ef-idm01 ~]# ipa-cert-fix
WARNING
ipa-cert-fix is intended for recovery when expired certificates prevent the normal operation of IPA. It should ONLY be used in such scenarios, and backup of the system, especially certificates and keys, is STRONGLY RECOMMENDED.
The following certificates will be renewed:
IPA IPA RA certificate: Subject: CN=IPA RA,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM Serial: 162 Expires: 2024-01-02 15:58:28
Enter "yes" to proceed: yes Proceeding. Renewed IPA IPA RA certificate: Subject: CN=IPA RA,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM Serial: 1341915142 Expires: 2026-02-03 20:18:20
Becoming renewal master. Restarting IPA
Note: Monitor the certmonger-initiated renewal of certificates after ipa-cert-fix and wait for its completion before any other administrative task.
The ipa-cert-fix command was successful [root@ef-idm01 ~]#
I checked the cert expiration several times yesterday, but it never updated on this server. I waited a full day to let certmonger do its thing, below is my result this morning.
[root@ef-idm01 ~]# ipactl restart Restarting Directory Service Restarting krb5kdc Service Restarting kadmin Service Restarting httpd Service Restarting ipa-custodia Service Restarting pki-tomcatd Service Restarting ipa-otpd Service ipa: INFO: The ipactl command was successful [root@ef-idm01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa: INFO: The ipactl command was successful [root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir' Request ID '20230530175932': status: MONITORING subject: CN=ef-idm01.production.efilm.comhttp://ef-idm01.production.efilm.com,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2025-05-30 10:59:53 PDT Request ID '20230530180022': status: MONITORING subject: CN=ef-idm01.production.efilm.comhttp://ef-idm01.production.efilm.com,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2025-05-30 11:00:30 PDT Request ID '20230530180438': status: NEED_CA subject: CN=IPA RA,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2024-01-02 07:58:28 PST [root@ef-idm01 ~]# ipa-cert-fix Nothing to do. The ipa-cert-fix command was successful [root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir' Request ID '20230530175932': status: MONITORING subject: CN=ef-idm01.production.efilm.comhttp://ef-idm01.production.efilm.com,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2025-05-30 10:59:53 PDT Request ID '20230530180022': status: MONITORING subject: CN=ef-idm01.production.efilm.comhttp://ef-idm01.production.efilm.com,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2025-05-30 11:00:30 PDT Request ID '20230530180438': status: NEED_CA subject: CN=IPA RA,O=PRODUCTION.EFILM.COMhttp://PRODUCTION.EFILM.COM expires: 2024-01-02 07:58:28 PST [root@ef-idm01 ~]#
How can I sort out this one remaining issue? Do I just make assign another server as the renewal master?
thanx
- grant
Grant Janssen via FreeIPA-users wrote:
When I upgraded the servers to EL8 (I rebuilt from scratch using the old hostnames), I had neglected to assign an IPA CA renewal master after the old “boss” was retired. This crime is of course it’s own punishment.
I found the documentation for handling this to actually be pretty good. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/htm...
fraser’s blog was also helpful (in confirming I executed this correctly) https://frasertweedale.github.io/blog-redhat/posts/2019-05-24-ipa-cert-fix.h...
I progressed through the other three IPA servers, but the last one still has a bad expiration on the CA cert.
[root@ef-idm01 ~]# date Wed Feb 14 07:08:38 PST 2024 [root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir' Request ID '20230530175932': status: MONITORING subject: CN=ef-idm01.production.efilm.com <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2025-05-30 10:59:53 PDT Request ID '20230530180022': status: MONITORING subject: CN=ef-idm01.production.efilm.com <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2025-05-30 11:00:30 PDT Request ID '20230530180438': status: NEED_CA subject: CN=IPA RA,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2024-01-02 07:58:28 PST [root@ef-idm01 ~]# ipa-cert-fix WARNING ipa-cert-fix is intended for recovery when expired certificates prevent the normal operation of IPA. It should ONLY be used in such scenarios, and backup of the system, especially certificates and keys, is STRONGLY RECOMMENDED. The following certificates will be renewed: IPA IPA RA certificate: Subject: CN=IPA RA,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> Serial: 162 Expires: 2024-01-02 15:58:28 Enter "yes" to proceed: yes Proceeding. Renewed IPA IPA RA certificate: Subject: CN=IPA RA,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> Serial: 1341915142 Expires: 2026-02-03 20:18:20 Becoming renewal master. Restarting IPA Note: Monitor the certmonger-initiated renewal of certificates after ipa-cert-fix and wait for its completion before any other administrative task. The ipa-cert-fix command was successful [root@ef-idm01 ~]#
I checked the cert expiration several times yesterday, but it never updated on this server. I waited a full day to let certmonger do its thing, below is my result this morning.
[root@ef-idm01 ~]#ipactl restart Restarting Directory Service Restarting krb5kdc Service Restarting kadmin Service Restarting httpd Service Restarting ipa-custodia Service Restarting pki-tomcatd Service Restarting ipa-otpd Service ipa: INFO: The ipactl command was successful [root@ef-idm01 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa: INFO: The ipactl command was successful [root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir' Request ID '20230530175932': status: MONITORING subject: CN=ef-idm01.production.efilm.com <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2025-05-30 10:59:53 PDT Request ID '20230530180022': status: MONITORING subject: CN=ef-idm01.production.efilm.com <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2025-05-30 11:00:30 PDT Request ID '20230530180438': status: NEED_CA subject: CN=IPA RA,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2024-01-02 07:58:28 PST [root@ef-idm01 ~]#ipa-cert-fix Nothing to do. The ipa-cert-fix command was successful [root@ef-idm01 ~]# getcert list | egrep '^Request|status:|subject:|expir' Request ID '20230530175932': status: MONITORING subject: CN=ef-idm01.production.efilm.com <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2025-05-30 10:59:53 PDT Request ID '20230530180022': status: MONITORING subject: CN=ef-idm01.production.efilm.com <http://ef-idm01.production.efilm.com>,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2025-05-30 11:00:30 PDT Request ID '20230530180438': status: NEED_CA subject: CN=IPA RA,O=PRODUCTION.EFILM.COM <http://PRODUCTION.EFILM.COM> expires: 2024-01-02 07:58:28 PST [root@ef-idm01 ~]#
How can I sort out this one remaining issue? Do I just make assign another server as the renewal master?
You only need to run ipa-cert-fix on the CA renewal master. Once that is done the RA Agent can be renewed on the machine and made available to the replicas.
Updated CA/RA certificates are stored in LDAP. Once replication is working then all replicas will pick up the necessary certificates from there. In fact you don't want to renew the CA/RA certificates per-machine. They all must be the same.
I'd need to see the full RA cert tracking to be able to tell what is going on but I'd suggest running ipa-server-upgrade first. That will fix any bad tracking. Then manually resubmit it with getcert if it is still in the NEED_CA state and it should pull the updated cert out of LDAP.
rob
this was definitely the hot tip. executing a server upgrade fixed everything for me.
thanx rob
well, I thought I was out of the woods, but I still have some issues. the services are running, but kinit gets me a ticket to nowhere.
"ipa: ERROR: No valid Negotiate header in server response"
grant@ef-idm01:~[20240220-14:36][#785]$ klist Ticket cache: KCM:555 Default principal: grant@PRODUCTION.EFILM.COM
Valid starting Expires Service principal 02/20/2024 14:36:12 02/21/2024 13:51:10 krbtgt/PRODUCTION.EFILM.COM@PRODUCTION.EFILM.COM grant@ef-idm01:~[20240220-14:36][#786]$ ipa server-find ipa: ERROR: No valid Negotiate header in server response grant@ef-idm01:~[20240220-14:36][#787]$ sudo systemctl status gssproxy.service ● gssproxy.service - GSSAPI Proxy Daemon Loaded: loaded (/usr/lib/systemd/system/gssproxy.service; disabled; vendor preset: disabled) Active: active (running) since Tue 2024-02-20 13:57:40 PST; 39min ago Process: 2158008 ExecStart=/usr/sbin/gssproxy -D (code=exited, status=0/SUCCESS) Main PID: 2158009 (gssproxy) Tasks: 6 (limit: 74714) Memory: 4.2M CGroup: /system.slice/gssproxy.service └─2158009 /usr/sbin/gssproxy -D
Feb 20 13:57:40 ef-idm01.production.efilm.com systemd[1]: gssproxy.service: Succeeded. Feb 20 13:57:40 ef-idm01.production.efilm.com systemd[1]: Stopped GSSAPI Proxy Daemon. Feb 20 13:57:40 ef-idm01.production.efilm.com systemd[1]: Starting GSSAPI Proxy Daemon... Feb 20 13:57:40 ef-idm01.production.efilm.com systemd[1]: Started GSSAPI Proxy Daemon. grant@ef-idm01:~[20240220-14:37][#788]$ sudo ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING httpd Service: RUNNING ipa-custodia Service: RUNNING pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa: INFO: The ipactl command was successful grant@ef-idm01:~[20240220-14:37][#789]$
I looked online for some references and it was suggested I replace the /var/lib/ipa/gssproxy/http.keytab The file looks OKAY to me though.
freeipa-users@lists.fedorahosted.org