Dungan, Scott A. via FreeIPA-users wrote:
Rob,
That worked!
No errors on healthcheck now. getcert list shows the expected output as well (9
certificates tracked just like on the other two servers).
I am not sure how to look for the LDAP entry for the HTTP service on ipa1 to remove the
old certificate blob.
I'd suggest using Apache Studio. It is a more user-friendly LDAP viewer
than ldapmodify/search.
What you're looking for is the userCertificate attribute for the HTTP
service entry. You'll need the dn for this. To get that easily you can run:
ipa service-show --all HTTP/ipa1.<blah>
The dn value will be displayed first.
If you look in /var/lib/ipa/certs/httpd.pem you'll see the
base64-encoded certificate in PEM format. A similar format is stored in
LDAP, just without the headers, footers and newlines.
userCertificate is a multi-valued which should have two certificates.
Remove the one that doesn't match /var/lib/ipa/certs/httpd.pem and you
should be good.
rob
Thanks,
-Scott
-----Original Message-----
From: Rob Crittenden <rcritten(a)redhat.com>
Sent: Thursday, December 2, 2021 12:33 PM
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
Cc: Dungan, Scott A. <sdungan(a)caltech.edu>
Subject: Re: [Freeipa-users] Re: ipa-healthcheck: ReplicationCheck ERROR
Try this. Move /var/lib/ipa/certs/httpd.crt to some place safe.
# getcert resubmit -i 20211130180109 -w -v
You'll be able to watch the progress. It should end up in MONITORING with a new
certificate.
Restart httpd.
So now you have a new, valid, known certificate for the service.
You'll need to modify the LDAP entry for this service to remove the original
certificate because it is unknown and likely to fail revocation by IPA tools as a result
(e.g. uninstallation). Figure out which of the userCertificate blobs is the old cert and
remove that value.
rob
Dungan, Scott A. via FreeIPA-users wrote:
> So, that certificate appears to be the newly installed web cert, signed by the
internal ipa CA for the ipa1 server itself:
>
> [root(a)ipa1.id.example.com ~]# getcert list -i 20211130180109 Number of
> certificates and requests being tracked: 9.
> Request ID '20211130180109':
> status: MONITORING
> stuck: no
> key pair storage:
type=FILE,location='/var/lib/ipa/private/httpd.key',pinfile='/var/lib/ipa/passwds/ipa1.id.example.com-443-RSA'
> certificate: type=FILE,location='/var/lib/ipa/certs/httpd.crt'
> CA: IPA
> issuer: CN=Certificate
Authority,O=ID.EXAMPLE.COM
> subject: CN=
ipa1.id.example.com,O=ID.EXAMPLE.COM
> expires: 2023-12-01 10:01:12 PST
> dns:
ipa1.id.example.com,ipa-ca.id.example.com
> principal name: HTTP/ipa1.id.example.com(a)ID.EXAMPLE.COM
> key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> eku: id-kp-serverAuth,id-kp-clientAuth
> pre-save command:
> post-save command: /usr/libexec/ipa/certmonger/restart_httpd
> track: yes
> auto-renew: yes
>
> -Scott
>
> -----Original Message-----
> From: Rob Crittenden <rcritten(a)redhat.com>
> Sent: Thursday, December 2, 2021 5:37 AM
> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
> Cc: Dungan, Scott A. <sdungan(a)caltech.edu>
> Subject: Re: [Freeipa-users] Re: ipa-healthcheck: ReplicationCheck
> ERROR
>
> So there was some data loss.
>
> Re-issuing a cert generally involves revoking the existing one but since that
isn't there, this could be interesting.
>
> What does the tracking for that cert look like? getcert list -i
> 20211130180109
>
> rob
>
> Dungan, Scott A. via FreeIPA-users wrote:
>> Update: after 20-30 minutes, the ca replication errors stopped, leaving only the
new IPACertRevocation error and the 404 code:
>>
>> [root(a)ipa1.id.example.com ~]# ipa-healthcheck --failures-only ...
>> ra.get_certificate(): Request failed with status 404: Non-2xx
>> response from CA REST API: 404. Certificate ID 0x13 not found (404) [
>> {
>> "source": "ipahealthcheck.ipa.certs",
>> "check": "IPACertRevocation",
>> "result": "ERROR",
>> "uuid": "c447d225-c712-4a77-b174-81f6ba008128",
>> "when": "20211202025711Z",
>> "duration": "3.825151",
>> "kw": {
>> "key": "20211130180109",
>> "serial": 19,
>> "error": "Certificate operation cannot be completed: Request
failed with status 404: Non-2xx response from CA REST API: 404. Certificate ID 0x13 not
found (404)",
>> "msg": "Request for certificate serial number {serial} in
request {key} failed: {error}"
>> }
>> }
>> ]
>
> So there was some data loss.
>
> Re-issuing a cert generally involves revoking the existing one but since that
isn't there, this could be interesting.
>
> What does the tracking for that cert look like? getcert list -i
> 20211130180109
>
> rob
>>
>> -----Original Message-----
>> From: Dungan, Scott A. via FreeIPA-users
>> <freeipa-users(a)lists.fedorahosted.org>
>> Sent: Wednesday, December 1, 2021 5:18 PM
>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>> Cc: Dungan, Scott A. <sdungan(a)caltech.edu>
>> Subject: [Freeipa-users] Re: ipa-healthcheck: ReplicationCheck ERROR
>>
>> Hi, Rob.
>>
>> I think I got ahead of myself and may have made things worse:
>>
>> Because servers iap2 and ipa3 are in agreement (ldif matches) and had no
healthcheck errors, and ipa1 did, I assumed that ipa1 needed to be re-initialized. The
small number of changes that would be lost on ipa1 was/is considered negligible. To
re-initialize ipa1, I ran the following:
>>
>> [root(a)ipa1.id.example.com ~]# ipa-csreplica-manage re-initialize --from
ipa3.id.example.com Directory Manager password:
>>
>> Update in progress, 7 seconds elapsed Update succeeded
>>
>> Looks good, but when I do a healthcheck again, now I get additional errors as
well as the original:
>>
>> [root(a)ipa1.id.example.com ~]# ipa-healthcheck --failures-only ...
>> ra.get_certificate(): Request failed with status 404: Non-2xx response from CA
REST API: 404. Certificate ID 0x13 not found (404 [
>> {
>> "source": "ipahealthcheck.ds.replication",
>> "check": "ReplicationCheck",
>> "result": "ERROR",
>> "uuid": "023f04e2-e8d8-459e-b2cb-bb516e30db07",
>> "when": "20211202005925Z",
>> "duration": "0.301820",
>> "kw": {
>> "key": "DSREPLLE0003",
>> "items": [
>> "Replication",
>> "Agreement"
>> ],
>> "msg": "The replication agreement (
catoipa2.id.example.com)
under \"o=ipaca\" is not in synchronization.\nStatus messa: error (18) can't
acquire replica (incremental update transient warning. backing off, will retry update
later.)"
>> }
>> },
>> {
>> "source": "ipahealthcheck.ds.replication",
>> "check": "ReplicationCheck",
>> "result": "ERROR",
>> "uuid": "6b7499ce-2138-415d-88ac-bf24f6f3a6db",
>> "when": "20211202005925Z",
>> "duration": "0.301851",
>> "kw": {
>> "key": "DSREPLLE0003",
>> "items": [
>> "Replication",
>> "Agreement"
>> ],
>> "msg": "The replication agreement (
catoipa3.id.example.com)
under \"o=ipaca\" is not in synchronization.\nStatus messa: error (18) can't
acquire replica (incremental update transient warning. backing off, will retry update
later.)"
>> }
>> },
>> {
>> "source": "ipahealthcheck.ipa.certs",
>> "check": "IPACertRevocation",
>> "result": "ERROR",
>> "uuid": "b02a8f85-985f-4b80-a099-fb3d415b2005",
>> "when": "20211202005935Z",
>> "duration": "2.663208",
>> "kw": {
>> "key": "20211130180109",
>> "serial": 19,
>> "error": "Certificate operation cannot be completed: Request
failed with status 404: Non-2xx response from CA REST API: 40 Certificate ID 0x13 not
found (404)",
>> "msg": "Request for certificate serial number {serial} in
request {key} failed: {error}"
>> }
>> }
>> ]
>>
>> -----Original Message-----
>> From: Rob Crittenden <rcritten(a)redhat.com>
>> Sent: Wednesday, December 1, 2021 11:50 AM
>> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
>> Cc: Dungan, Scott A. <sdungan(a)caltech.edu>
>> Subject: Re: [Freeipa-users] ipa-healthcheck: ReplicationCheck ERROR
>>
>> Dungan, Scott A. via FreeIPA-users wrote:
>>> We have 3 ipa servers, one of which is throwing an ERROR condition
>>> during ipa-healthcheck for the "ReplicationCheck" test.
>>> Ipa-healthcheck shows no errors when run from the other two replicas.
>>> Looking back at the logs, it appears this started about ten days
>>> ago, so it is not a transient issue as the output suggests:
>>>
>>>
>>>
>>> [root(a)ipa1.id.example.com]# ipa-healthcheck --failures-only
>>>
>>>
>>>
>>> [
>>>
>>> {
>>>
>>> "source": "ipahealthcheck.ds.replication",
>>>
>>> "check": "ReplicationCheck",
>>>
>>> "result": "ERROR",
>>>
>>> "uuid": "2b971ca3-678e-4c26-86a0-5b352027e7e8",
>>>
>>> "when": "20211201180013Z",
>>>
>>> "duration": "0.687812",
>>>
>>> "kw": {
>>>
>>> "key": "DSREPLLE0003",
>>>
>>> "items": [
>>>
>>> "Replication",
>>>
>>> "Agreement"
>>>
>>> ],
>>>
>>> "msg": "The replication agreement
(
catoipa2.id.example.com)
>>> under \"o=ipaca\" is not in synchronization.\nStatus message:
error
>>> (18) can't acquire replica (incremental update transient warning.
>>> backing off, will retry update later.)"
>>>
>>> }
>>>
>>> },
>>>
>>> {
>>>
>>> "source": "ipahealthcheck.ds.replication",
>>>
>>> "check": "ReplicationCheck",
>>>
>>> "result": "ERROR",
>>>
>>> "uuid": "99436870-bc98-4ce8-84b1-c0b0806945c8",
>>>
>>> "when": "20211201180013Z",
>>>
>>> "duration": "0.687829",
>>>
>>> "kw": {
>>>
>>> "key": "DSREPLLE0003",
>>>
>>> "items": [
>>>
>>> "Replication",
>>>
>>> "Agreement"
>>>
>>> ],
>>>
>>> "msg": "The replication agreement
(
catoipa3.id.example.com)
>>> under \"o=ipaca\" is not in synchronization.\nStatus message:
error
>>> (18) can't acquire replica (incremental update transient warning.
>>> backing off, will retry update later.)"
>>>
>>> }
>>>
>>> }
>>>
>>> ]
>>>
>>>
>>>
>>> 389-ds error logs show a slew of these:
>>>
>>>
>>>
>>> [30/Nov/2021:23:41:35.277399980 -0800] - ERR - NSMMReplicationPlugin
>>> - send_updates - agmt="cn=caToipa3.id.example.com" (ipa2:389):
>>> Missing data encountered. If the error persists the replica must be
reinitialized.
>>>
>>> [30/Nov/2021:23:41:38.288003253 -0800] - ERR -
>>> agmt="cn=caToipa3.id.example.com" (ipa3:389) - clcache_load_buffer
-
>>> Can't locate CSN 6197e149000000060000 in the changelog (DB rc=-30988).
>>> If replication stops, the consumer may need to be reinitialized.
>>>
>>> [30/Nov/2021:23:41:38.289713999 -0800] - ERR - NSMMReplicationPlugin
>>> - send_updates - agmt="cn=caToipa3.id.example.com" (ipa3:389):
>>> Missing data encountered. If the error persists the replica must be
reinitialized.
>>>
>>>
>>>
>>> That would seem to suggest running a "ipa-replica-manage
>>> re-initialize --from $SERVER_TO_PULL_FROM" may resolve the issue,
>>> but before we try that, is there anything else we should look at?
>>
>> You use ipa-csreplica-manage to manage the CA replication agreements.
>> But yes, it looks like you need to re-initialize some of them.
>>
>> I'd suggest dump the ldif of the two to be re-inited and see if there are any
entries there not recorded in the one you will re-init from to see if there is any
potential data loss.
>>
>> rob
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to
>> freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct:
>>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines:
>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedo
>> r
ahosted.org Do not reply to spam on the list, report it:
>>
https://pagure.io/fedora-infrastructure
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> To unsubscribe send an email to
>> freeipa-users-leave(a)lists.fedorahosted.org
>> Fedora Code of Conduct:
>>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List Guidelines:
>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List Archives:
>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedo
>> r
ahosted.org Do not reply to spam on the list, report it:
>>
https://pagure.io/fedora-infrastructure
>>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
>
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedor
>
ahosted.org Do not reply to spam on the list, report it:
>
https://pagure.io/fedora-infrastructure
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure