Hello,

I have a strange issue regarding acme service.
My acme certificates fail to renew. `ipa-acme-manage status`fails with error:
Failed to authenticate to CA REST API
The ipa-acme-manage command failed.

certbot client fails with error "Failed to renew certificate office.empire.lan with error: <Response [404]>"

$ ipa cert-show 49
 Issuing CA: ipa
 Certificate: "The certificate content"
 Subject: CN=office.empire.lan
 Subject DNS name: office.empire.lan
 Issuer: CN=Certificate Authority,O=EMPIRE.LAN
 Not Before: Sun Dec 24 14:05:50 2023 UTC
 Not After: Sat Mar 23 14:05:50 2024 UTC
 Serial number: 49
 Serial number (hex): 0x31
 Revoked: False

So last successful renewal was on Dec 24th. Since then I have not really done anything appart updating.
I don't see any issue in ipaupgrade.log


I am running on centos stream 9
idm-jss.x86_64                                                                                                       5.5.0-1.el9
idm-jss-tomcat.x86_64                                                                                                5.5.0-1.el9
idm-ldapjdk.noarch                                                                                                   5.5.0-1.el9
idm-pki-acme.noarch                                                                                                  11.5.0-1.el9
idm-pki-base.noarch                                                                                                  11.5.0-1.el9
idm-pki-ca.noarch                                                                                                    11.5.0-1.el9
idm-pki-java.noarch                                                                                                  11.5.0-1.el9
idm-pki-kra.noarch                                                                                                   11.5.0-1.el9
idm-pki-server.noarch                                                                                                11.5.0-1.el9
idm-pki-tools.x86_64                                                                                                 11.5.0-1.el9
ipa-client.x86_64                                                                                                    4.11.0-9.el9
ipa-client-common.noarch                                                                                             4.11.0-9.el9
ipa-common.noarch                                                                                                    4.11.0-9.el9
ipa-healthcheck.noarch                                                                                               0.16-2.el9
ipa-healthcheck-core.noarch                                                                                          0.16-2.el9
ipa-selinux.noarch                                                                                                   4.11.0-9.el9
ipa-server.x86_64                                                                                                    4.11.0-9.el9
ipa-server-common.noarch                                                                                             4.11.0-9.el9
ipa-server-dns.noarch                                                                                                4.11.0-9.el9

I have followed closely the update on centos stream 9

Running  `ipa-acme-manage status` with the -d switch gives me
ipapython.ipaldap: DEBUG: retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-EMPIRE-LAN.socket conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7f123c07e2e0>
ipaserver.masters: DEBUG: Discovery: available servers for service 'CA' are ipa-server-01.empire.lan, ipa-server-02.empire.lan
ipaserver.masters: DEBUG: Discovery: using ipa-server-01.empire.lan for 'CA' service
ipapython.dogtag: DEBUG: request POST https://ipa-server-01.empire.lan:8443/acme/login
ipapython.dogtag: DEBUG: request body ''
ipapython.dogtag: DEBUG: response status 404
ipapython.dogtag: DEBUG: response headers Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 765
Date: Thu, 28 Mar 2024 10:00:59 GMT


ipapython.dogtag: DEBUG: response body (decoded): b'<!doctype html><html lang="en"><head><title>HTTP Status 404 \xe2\x80\x93 Not Found</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 \xe2\x80\x93 Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> The requested resource [&#47;acme&#47;login] is not available</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat/9.0.62</h3></body></html>'
ipapython.admintool: DEBUG:   File "/usr/lib/python3.9/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
  File "/usr/lib/python3.9/site-packages/ipaserver/install/ipa_acme_manage.py", line 403, in run
    with state as ca_api:
  File "/usr/lib/python3.9/site-packages/ipaserver/install/ipa_acme_manage.py", line 103, in __enter__
    raise errors.RemoteRetrieveError(

ipapython.admintool: DEBUG: The ipa-acme-manage command failed, exception: RemoteRetrieveError: Failed to authenticate to CA REST API
ipapython.admintool: ERROR: Failed to authenticate to CA REST API
ipapython.admintool: ERROR: The ipa-acme-manage command failed.


So it looks like the acme subsystem is not started. But logs for the acme subsystem in /var/log/pki/pki-tomcat/acme/debug.2024-03-28.log don't show any issue. (see attached log)

How can I go further in troubleshooting/fixing this issue?

Thanks