Hi Christopher,
I agree with Rob that replication issue is the most likely cause.
If there were replication issues, depending on your topology there
may be serial/request ID range conflicts too. But the most critical
issue is the about-to-expire certificate.
A couple of quick points/questions:
- The expiring certificate is the Server-Cert, other CA replicas
will have different Server-Certs so they will continue to
function.
- Are there any other certs on this replica, or others, that are
close to expiry?
Now to the error:
"http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit"
replied: Record not found
It is not clear exactly what record is missing. It is likely either
the certificate record, or its corresponding request record. The
Dogtag debug log (/var/log/pki/pki-tomcat/ca/debug) may reveal more.
In any case, have a hunt for
cn=268304422,ou=certificateRepository,ou=ca,o=ipaca
If found, in the entry there should be an attribute:
metaInfo: requestId:<N>
for some value of <N>. Now also look for the entry:
cn=<N>,ou=ca,ou=requests,o=ipaca
If any of these entries can be found on other replicas but not the
database on the replica where the cert is expiring, you can manually
export/import them, and it might solve the issue.
Otherwise, I recall a recent issue where the workaround was to make
the Certmonger renewal helper do a "new issuance" rather than a
"renewal"-based operation against the Dogtag CA. This could help in
your situation too. I am not sure whether or where the steps were
recorded so Rob, Florence - do you know?
Anyhow it is possible I have gone down the garden path so it would
really help to see the relevant portion of the Dogtag debug log.
(Be aware Dogtag timestamps are in local time, when you are looking
for the relevant output).
Cheers,
Fraser
On Tue, Dec 04, 2018 at 09:47:11PM -0500, Christopher Young via FreeIPA-users wrote:
Another thing I notice that confuses me... (see attached)
Is it normal to have this many certificate with the same Subject for
an IPA server? I'm wondering if somewhere along it renewed and yet
didn't update locally or something. I'm really not sure what's going
on here, but what I'm confused about is IF I wanted to generate a new
server for the IPA server, how would I go about doing that in a manner
where the certificate would have all the right attributes? (and would
I want to do that?)
Sorry for all the questions. I'm figuring the pieces out as I go.
On Tue, Dec 4, 2018 at 9:04 PM Christopher Young <mexigabacho(a)gmail.com> wrote:
>
> Output:
> ----
> [root@orldc-prod-ipa01 alias]# ipa-csreplica-manage list -v
> `hostname`.passur.local
> Directory Manager password:
>
> orldc-prod-ipa02.passur.local
> last init status: None
> last init ended: 1970-01-01 00:00:00+00:00
> last update status: Error (-1) Problem connecting to replica - LDAP
> error: Can't contact LDAP server (connection error)
> last update ended: 1970-01-01 00:00:00+00:00
> ----
>
> Granted, it's replication partner (orldc-prod-ipa02) is the one that I
> mentioned has the issue starting at this point. So, that likely has
> something to do with this output. Having said that, I'm not quite
> sure what I should do here. I have definitely been issuing certs from
> this system. One note is that the 'hostname' command on my systems
> return only the short hostname. I'm not sure if that would be an
> issue or not, but it is worth noting. (To add 'domainname' does show
> the proper domainname for the system and what-not.
>
> Any ideas on the best way to fix just this host? I don't mind the
> idea of removing the CA replicas on the others after fixing this and
> re-replicating or anything. I just want to get this functioning
> before something exprires and I end up in a really bad spot (which is
> sadly only a day away!). (Why, oh why, do we always 'find' these
> type of problems under a time crunch in this business?) :)
>
>
>
>
> On Tue, Dec 4, 2018 at 5:57 PM Rob Crittenden <rcritten(a)redhat.com> wrote:
> >
> > Christopher Young via FreeIPA-users wrote:
> > > Yeah. I definitely lost on this one at this point. As far as I can
> > > tell, SOMEHOW I'm missing these certs in the directory? Does that
> > > sound right?
> > >
> > > How would one go about making sure is corrected? I'm guess I'd
need
> > > to regenerate some type of certificate on the IPA host, but I'm
afraid
> > > of breaking things worse. I have one more day before this one
> > > expires, so I'm trying to troubleshoot and fix it before then. This
> > > all started when I noticed that another IPA server/replica failed to
> > > restart. I think these two issues are related, but right now, my
> > > users are functional with just the 'ipa01' system (which has this
> > > ca-error' issue and the cert not found. I'm afraid to restart
> > > anything on that system because of that.
> > >
> > > I'm still reading and trying to understand and put the pieces
> > > together, however I'm worried about this issue.
> >
> > It sounds like one of your CA's is not replicating. You can use
> > ipa-csreplica-manage list -v `hostname` on each CA master to get the status.
> >
> > Any given replica can only store so much data to replicate so depending
> > on how long they have been disconnected could impact whether this is
> > easily recoverable.
> >
> > Given that, on any master it should always use its own CA (if there is
> > one) when issuing certs so this is a bit strange.
> >
> > rob
> >
> > >
> > > Anyway, if anyone has any thoughts or tips here, I'd really
appreciate
> > > it as I feel lost at this exact moment.
> > > On Tue, Dec 4, 2018 at 2:33 PM Christopher Young
<mexigabacho(a)gmail.com> wrote:
> > >>
> > >> IPA 4.5.4 (has been upgraded for years just to understand that there
> > >> is a history)
> > >> This system (ipa01) is the renewal master (in case that matters)
> > >>
> > >> I'm getting the following error on 'getcert'. My gut
tells me this is
> > >> kinda a big deal. :) I really could use some help figuring this one
> > >> out as I'm not the most CA-versed. I have been learning quite a
bit
> > >> reading some of the blogs, but there's definitely alot of
ignorance of
> > >> the details on my part.
> > >>
> > >> The error:
> > >> -----------
> > >> [root@orldc-prod-ipa01 log]# getcert list | grep -A12 -B1 error
> > >> status: MONITORING
> > >> ca-error: Server at
> > >>
"http://orldc-prod-ipa01.passur.local:8080/ca/ee/ca/profileSubmit"
> > >> replied: Record not found
> > >> stuck: no
> > >> key pair storage:
> > >>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> > >> cert-pki-ca',token='NSS Certificate DB',pin set
> > >> certificate:
> > >>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
> > >> cert-pki-ca',token='NSS Certificate DB'
> > >> CA: dogtag-ipa-ca-renew-agent
> > >> issuer: CN=Certificate Authority,O=PASSUR.LOCAL
> > >> subject: CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL
> > >> expires: 2018-12-06 21:43:50 UTC
> > >> key usage:
> > >> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> > >> eku: id-kp-serverAuth,id-kp-clientAuth,id-kp-emailProtection
> > >> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
> > >> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> > >> "Server-Cert cert-pki-ca"
> > >> track: yes
> > >> -----------
> > >>
> > >>
> > >> If I look at the cert referenced locally in the NSS DB:
> > >> ------
> > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias
> > >> -f /etc/httpd/alias/pwdfile.txt
> > >>
> > >> Certificate Nickname Trust
Attributes
> > >>
SSL,S/MIME,JAR/XPI
> > >>
> > >> Server-Cert cert-pki-ca u,u,u
> > >> auditSigningCert cert-pki-ca u,u,Pu
> > >> caSigningCert cert-pki-ca
CTu,Cu,Cu
> > >> subsystemCert cert-pki-ca u,u,u
> > >> ocspSigningCert cert-pki-ca u,u,u
> > >> ------
> > >> [root@orldc-prod-ipa01 log]# certutil -L -d /etc/pki/pki-tomcat/alias
> > >> -f /etc/httpd/alias/pwdfile.txt -n 'Server-Cert cert-pki-ca' |
grep
> > >> "Subject:\|Serial"
> > >> Serial Number: 268304422 (0xffe0026)
> > >> Subject:
"CN=orldc-prod-ipa01.passur.local,O=PASSUR.LOCAL"
> > >> -----
> > >> [root@orldc-prod-ipa01 log]# ipa cert-find --min-serial-number
> > >> 268304422 --max-serial-number 268304423
> > >> ----------------------
> > >> 0 certificates matched
> > >> ----------------------
> > >> -----
> > >>
> > >> I'm trying to figure out how to find this certificate. And IF
somehow
> > >> it is wrong or missing, how do I fix such a scenario?
> > >>
> > >> Any help here is always appreciated! Unfortunately, I'm running
out
> > >> of time based on the expiration date I see on 'getcert'.
I'm not sure
> > >> of the ramifications, but this seems pretty critical on the surface.
> > >>
> > >> Thanks again for any help and direction!
> > >>
> > >> -- Chris
> > > _______________________________________________
> > > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> > > To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
> > > Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> > > List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
> > >
> >
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...