Hi,
On Wed, Jun 28, 2023 at 4:45 PM Rob Crittenden via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org> wrote:
Jernej Jakob via FreeIPA-users wrote:
> I've been trying to debug this for the last couple of days. I can't
> find what's wrong. I found that another client whose cert also expired
> on 2023-06-07 was in the same SUBMITTING state. The same exact
> conditions. Same exact OS, Ubuntu 20.04 LTS. certmonger package is
> up-to-date.
>
> I increased certmonger debug level to 6.
I'd suggest 3. 6 will include a lot of unnecessary dbus noise.
> I also changed in /var/lib/certmonger/cas/20141205060547-3
> ca_external_helper=/usr/lib/certmonger/dogtag-ipa-renew-agent-submit
> to
> ca_external_helper=/usr/lib/certmonger/dogtag-ipa-renew-agent-submit -v
That is the wrong certmonger CA. You want the IPA CA helper. This one is
for renewing the CA certificates itself. And yes CA here is an
overloaded term.
> The logs are attached.
>
> What I find weird is that on line 180:
> GET "http://redacted.example.com:9180/ca/ee/ca/profileList?xml=true"
> this is the client's address, not the ipa-ca server's.
Can you check the output of "getcert list-cas -c IPA"? What is the
content of /etc/certmonger/certmonger.conf?
Does /etc/ipa/default.conf contain a line with dogtag_version = <some
value> ?
flo
Perhaps check the values in /etc/ipa/default.conf.
> I found these that may or may not be related
>
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955530
>
https://salsa.debian.org/freeipa-team/certmonger/-/blob/debian/0.79.9-2/d...
>
https://bugs.launchpad.net/ubuntu/+source/certmonger/+bug/1875227
>
> Any ideas on what to do next?
>
> On Fri, 16 Jun 2023 15:38:46 +0200
> Jernej Jakob <jernej.jakob(a)gmail.com> (by way of Jernej Jakob
> <jernej.jakob(a)abak.si>) wrote:
>
>> Hi, I have a client whose host certificate expired on 2023-06-07.
>> Today I logged into the FreeIPA webui and went to the certificates page
>> which was very slow to load. I had this problem before when there was
>> one host (a different one) stuck in a certificate request loop, so I
>> immediately suspected the same thing happened again. Sure enough, there
>> are >3000 certificates for this host listed in IPA.
>> Running 'getcert list' on the host shows:
>>
>> Number of certificates and requests being tracked: 1.
>> Request ID '20190703221417':
>> status: SUBMITTING
>> stuck: no
>> key pair storage:
type=NSSDB,location='/etc/ipa/nssdb',nickname='Local IPA
host',token='NSS
Certificate DB',pinfile='/etc/ipa/nssdb/pwdfile.txt'
>> certificate:
type=NSSDB,location='/etc/ipa/nssdb',nickname='Local IPA
host',token='NSS
Certificate DB'
>> CA: IPA
>> issuer: CN=Certificate
Authority,O=EXAMPLE.COM
>> subject:
CN=redacted.example.com,O=EXAMPLE.com
>> expires: 2023-06-07 00:14:30 CEST
>> dns:
redacted.example.com
>> principal name: host/redacted.example.com(a)EXAMPLE.COM
>> key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> eku: id-kp-serverAuth,id-kp-clientAuth
>> pre-save command:
>> post-save command:
>> track: yes
>> auto-renew: yes
>>
>> I stopped certmonger to stop the loop. Then checked if the problem was
>> the same as on that other host some time ago, but it was not.
>> I saw one error in syslog (this turned out to not be the issue):
>>
>> "Server at
https://ipa1.example.com/ipa/xml failed request, will
retry: -504 (libcurl failed to execute the HTTP POST transaction,
explaining: Could not resolve host: ipa1.example.com)."
>>
>> That is a server that has been shut down. It was the first server I
>> installed IPA onto, which had a replica ipa2. That was on CentOS 7.
>> This year I migrated to Rocky 8 by adding two new clients, ipa3 and
>> ipa4, following the official procedure by promoting each client to a
>> replica, set ipa3 as CA renewal master, and so on. Then after that was
>> done I removed the old servers from replication, shut down IPA services,
>> unenrolled them and updated all SRV records in DNS. I forgot about
>> A/AAAA ipa-ca, which was left pointing at two addresses, that of ipa1
>> and ipa3.
>>
>> It's weird becase it can obviously contact and obtain a certificate
>> from ipa3, whose logs contain these successful issuances. It could
>> contact it on 2023-06-07, when the ipa-ca record was still pointing to
>> both the ipa1 and ipa3 servers.
>> On 2023-06-09 I dist-upgraded both ipa3/4 servers, at which time I also
>> found the incorrect ipa-ca A/AAAA records (coincidentally as I was
>> configuring a smart card CA certificate), so I corrected it. But the
>> failing certmonger client still kept failing which makes me believe
>> it was not the fault of the incorrect ipa-ca records.
>>
>> I searched around on the "redacted.example.com" for ipa1. It's not
in
>> /var/lib/certmonger/requests/20190703221417 (the failing request)
>> but it is in /etc/ipa/default.conf:
>>
>> server =
ipa1.example.com
>> xmlrpc_uri =
https://ipa1.example.com/ipa/xml
>>
>> I changed these values to ipa3 now, started certmonger and resubmitted
>> the request. Unfortunately no change. That libcurl error for ipa1 is
>> gone now:
>>
>> Jun 16 15:28:01 redacted certmonger[2091940]: 2023-06-16 15:28:01
[2091940] Token is named "NSS Generic Crypto Services", not "NSS
Certificate DB", skipping.
>> Jun 16 15:28:01 redacted certmonger[2091940]: 2023-06-16 15:28:01
[2091940] Unable to initialize NSS.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_REQ_SUBJECT" to "CN=redacted.example.com,O=
EXAMPLE.COM" for child.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_REQ_PRINCIPAL" to "host/
redacted.example.com(a)EXAMPLE.COM" for child.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_OPERATION" to "SUBMIT" for child.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_CSR" to "-----BEGIN NEW CERTIFICATE
REQUEST-----
>> Jun 16 15:28:01 redacted certmonger[2091941]: ...redacted..." for child.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_SPKAC" to ...redacted...
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_SPKI" to ...redacted...
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_LOCAL_CA_DIR" to
"/var/lib/certmonger/local"
for child.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_KEY_TYPE" to "RSA" for child.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_CA_NICKNAME" to "IPA" for child.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Setting "CERTMONGER_CERTIFICATE" to "-----BEGIN
CERTIFICATE-----
>> Jun 16 15:28:01 redacted certmonger[2091941]: ...redacted..." for child.
>> Jun 16 15:28:01 redacted certmonger[2091941]: 2023-06-16 15:28:01
[2091941] Redirecting stdin and stderr to /dev/null, leaving stdout open
for child "/usr/lib/certmonger/ipa-submit".
>>
>> At this point it just keeps repeating the same over and over again.
>>
>> I would appreciate any help. Thanks.
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue