I've been trying to debug this for the last couple of days. I
can't
find what's wrong. I found that another client whose cert also expired
on 2023-06-07 was in the same SUBMITTING state. The same exact
conditions. Same exact OS, Ubuntu 20.04 LTS. certmonger package is
up-to-date.
I increased certmonger debug level to 6.
I'd suggest 3. 6 will include a lot of unnecessary dbus noise.
I also changed in /var/lib/certmonger/cas/20141205060547-3
ca_external_helper=/usr/lib/certmonger/dogtag-ipa-renew-agent-submit
to
ca_external_helper=/usr/lib/certmonger/dogtag-ipa-renew-agent-submit -v
That is the wrong certmonger CA. You want the IPA CA helper. This one is
for renewing the CA certificates itself. And yes CA here is an
overloaded term.
The logs are attached.
What I find weird is that on line 180:
GET "http://redacted.example.com:9180/ca/ee/ca/profileList?xml=true"
this is the client's address, not the ipa-ca server's.
I found these that may or may not be related
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955530
https://salsa.debian.org/freeipa-team/certmonger/-/blob/debian/0.79.9-2/d...
https://bugs.launchpad.net/ubuntu/+source/certmonger/+bug/1875227
Any ideas on what to do next?
On Fri, 16 Jun 2023 15:38:46 +0200
Jernej Jakob <jernej.jakob(a)gmail.com> (by way of Jernej Jakob
<jernej.jakob(a)abak.si>) wrote:
> Hi, I have a client whose host certificate expired on 2023-06-07.
> Today I logged into the FreeIPA webui and went to the certificates page
> which was very slow to load. I had this problem before when there was
> one host (a different one) stuck in a certificate request loop, so I
> immediately suspected the same thing happened again. Sure enough, there
> are >3000 certificates for this host listed in IPA.
> Running 'getcert list' on the host shows:
>
> Number of certificates and requests being tracked: 1.
> Request ID '20190703221417':
> status: SUBMITTING
> stuck: no
> key pair storage:
type=NSSDB,location='/etc/ipa/nssdb',nickname='Local IPA
host',token='NSS Certificate DB',pinfile='/etc/ipa/nssdb/pwdfile.txt'
> certificate: type=NSSDB,location='/etc/ipa/nssdb',nickname='Local
IPA host',token='NSS Certificate DB'
> CA: IPA
> issuer: CN=Certificate
Authority,O=EXAMPLE.COM
> subject:
CN=redacted.example.com,O=EXAMPLE.com
> expires: 2023-06-07 00:14:30 CEST
> dns:
redacted.example.com
> principal name: host/redacted.example.com(a)EXAMPLE.COM
> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command:
> post-save command:
> track: yes
> auto-renew: yes
>
> I stopped certmonger to stop the loop. Then checked if the problem was
> the same as on that other host some time ago, but it was not.
> I saw one error in syslog (this turned out to not be the issue):
>
> "Server at
https://ipa1.example.com/ipa/xml failed request, will retry: -504
(libcurl failed to execute the HTTP POST transaction, explaining: Could not resolve host:
ipa1.example.com)."
>
> That is a server that has been shut down. It was the first server I
> installed IPA onto, which had a replica ipa2. That was on CentOS 7.
> This year I migrated to Rocky 8 by adding two new clients, ipa3 and
> ipa4, following the official procedure by promoting each client to a
> replica, set ipa3 as CA renewal master, and so on. Then after that was
> done I removed the old servers from replication, shut down IPA services,
> unenrolled them and updated all SRV records in DNS. I forgot about
> A/AAAA ipa-ca, which was left pointing at two addresses, that of ipa1
> and ipa3.
>
> It's weird becase it can obviously contact and obtain a certificate
> from ipa3, whose logs contain these successful issuances. It could
> contact it on 2023-06-07, when the ipa-ca record was still pointing to
> both the ipa1 and ipa3 servers.
> On 2023-06-09 I dist-upgraded both ipa3/4 servers, at which time I also
> found the incorrect ipa-ca A/AAAA records (coincidentally as I was
> configuring a smart card CA certificate), so I corrected it. But the
> failing certmonger client still kept failing which makes me believe
> it was not the fault of the incorrect ipa-ca records.
>
> I searched around on the "redacted.example.com" for ipa1. It's not in
> /var/lib/certmonger/requests/20190703221417 (the failing request)
> but it is in /etc/ipa/default.conf:
>
> server =
ipa1.example.com
> xmlrpc_uri =
https://ipa1.example.com/ipa/xml
>
> I changed these values to ipa3 now, started certmonger and resubmitted
> the request. Unfortunately no change. That libcurl error for ipa1 is
> gone now:
>
> Jun 16 15:28:01 redacted certmonger[2091940]: 2023-06-16 15:28:01 [2091940] Token is
named "NSS Generic Crypto Services", not "NSS Certificate DB",
skipping.
> Jun 16 15:28:01 redacted certmonger[2091940]: 2023-06-16 15:28:01 [2091940] Unable to
initialize NSS.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_REQ_SUBJECT" to "CN=redacted.example.com,O=EXAMPLE.COM"
for child.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_REQ_PRINCIPAL" to "host/redacted.example.com(a)EXAMPLE.COM"
for child.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_OPERATION" to "SUBMIT" for child.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_CSR" to "-----BEGIN NEW CERTIFICATE REQUEST-----
> Jun 16 15:28:01 redacted certmonger[2091941]: ...redacted..." for child.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_SPKAC" to ...redacted...
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_SPKI" to ...redacted...
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_LOCAL_CA_DIR" to "/var/lib/certmonger/local" for child.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_KEY_TYPE" to "RSA" for child.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_CA_NICKNAME" to "IPA" for child.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941] Setting
"CERTMONGER_CERTIFICATE" to "-----BEGIN CERTIFICATE-----
> Jun 16 15:28:01 redacted certmonger[2091941]: ...redacted..." for child.
> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01 [2091941]
Redirecting stdin and stderr to /dev/null, leaving stdout open for child
"/usr/lib/certmonger/ipa-submit".
>
> At this point it just keeps repeating the same over and over again.
>
> I would appreciate any help. Thanks.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue