A relatively good news:
The current error (Insufficient access: Principal 'HTTP/spinque04.hq.spinque.com@HQ.SPINQUE.COM' is not permitted to use CA '.' with profile 'caIPAserviceCert' for certificate issuance.) might not be due to the package upgrade.

I looked at the journal of 16 Feb 2017 (28 days before the expiration date): certmonger correctly tries to renew the certificate but fails with the exact same error as I have now. So this explains why I ended up with an expired certificate in the first place. 

So hopefully I'm back to the original issue that caused all this. Any help is highly appreciated.


On Wed, 7 Jun 2017 at 19:15 Rob Crittenden <rcritten@redhat.com> wrote:
Roberto Cornacchia via FreeIPA-users wrote:
> Sorry for accidentally dropping freeipa-users.
>
> I was impatient so went back in time before your answer, but I did chose
> a good date
>
> Before this, I had the following two entries with an expired date:
>
> Request ID '20150316184508':
> status: NEED_TO_SUBMIT
> ca-error: Error setting up ccache for "host" service on client using
> default keytab: Cannot contact any KDC for requested realm.
>
> Request ID '20150316184529':
> status: CA_UNREACHABLE
> ca-error: Error setting up ccache for "host" service on client using
> default keytab: Cannot contact any KDC for requested realm.
>
> After restarting certmonger, I have:
>
> Request ID '20150316184508':
>         status: MONITORING
>         ca-error: Server at https://spinque04.hq.spinque.com/ipa/xml
> denied our request, giving up: 2100 (RPC failed at server.  Insufficient
> access: Principal 'ldap/spinque04.hq.spinque.com@HQ.SPINQUE.COM
> <mailto:spinque04.hq.spinque.com@HQ.SPINQUE.COM>' is not permitted to
> use CA '.' with profile 'caIPAserviceCert' for certificate issuance.).
>
> Request ID '20150316184529':
>         status: MONITORING
>         ca-error: Server at https://spinque04.hq.spinque.com/ipa/xml
> denied our request, giving up: 2100 (RPC failed at server.  Insufficient
> access: Principal 'HTTP/spinque04.hq.spinque.com@HQ.SPINQUE.COM
> <mailto:spinque04.hq.spinque.com@HQ.SPINQUE.COM>' is not permitted to
> use CA '.' with profile 'caIPAserviceCert' for certificate issuance.).

I think this is a side-effect of updating the packages with expired
certs (you are about half upgraded right now). The new CA ACL rules
don't seem to have been applied. I'm not sure what the safest course in,
maybe Flo or Fraser know.

rob

>
> The journal shows continuous Tomcat errors such as this:
>
> Mar 01 00:09:28 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> named-pkcs11[5692]: validating
> docs.oracle.com/CNAME <http://docs.oracle.com/CNAME>: bad cache hit
> (oracle.com/DS <http://oracle.com/DS>)
> Mar 01 00:09:28 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> named-pkcs11[5692]: broken trust chain
> resolving 'docs.oracle.com/A/IN <http://docs.oracle.com/A/IN>': 8.8.8.8#53
> Mar 01 00:09:28 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> named-pkcs11[5692]: validating
> docs.oracle.com/CNAME <http://docs.oracle.com/CNAME>: bad cache hit
> (oracle.com/DS <http://oracle.com/DS>)
> Mar 01 00:09:28 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> named-pkcs11[5692]: broken trust chain
> resolving 'docs.oracle.com/AAAA/IN <http://docs.oracle.com/AAAA/IN>':
> 8.8.8.8#53
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: Mar 01, 2017 12:09:32 AM
> org.apache.catalina.core.ContainerBase backgroundProcess
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: WARNING: Exception
> processing realm com.netscape.cms.tomcat.ProxyRealm@4077b502 background
> process
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]:
> java.lang.NullPointerException
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: at
> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:109)
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: at
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1154)
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: at
> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5697)
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1377)
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1381)
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
> Mar 01 00:09:32 spinque04.hq.spinque.com
> <http://spinque04.hq.spinque.com> server[5912]: at
> java.lang.Thread.run(Thread.java:745)
>
> On Wed, 7 Jun 2017 at 16:22 Rob Crittenden <rcritten@redhat.com
> <mailto:rcritten@redhat.com>> wrote:
>
>     Adding freeipa-users back.
>
>     Roberto Cornacchia wrote:
>     > Just to confirm, this is indeed the expired certificate
>     >
>     > $ getcert list -d /etc/httpd/alias -n Server-Cert
>     > Number of certificates and requests being tracked: 8.
>     > Request ID '20150316184529':
>     > status: CA_UNREACHABLE
>     > ca-error: Error setting up ccache for "host" service on client using
>     > default keytab: Cannot contact any KDC for requested realm.
>     > stuck: no
>     > key pair storage:
>     >
>     type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>     > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>     > certificate:
>     >
>     type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>     > Certificate DB'
>     > CA: IPA
>     > issuer: CN=Certificate Authority,O=HQ.SPINQUE.COM
>     <http://HQ.SPINQUE.COM> <http://HQ.SPINQUE.COM>
>     > subject: CN=spinque04.hq.spinque.com <http://spinque04.hq.spinque.com>
>     > <http://spinque04.hq.spinque.com>,O=HQ.SPINQUE.COM
>     <http://HQ.SPINQUE.COM> <http://HQ.SPINQUE.COM>
>     > expires: 2017-03-16 18:45:29 UTC
>     > key usage:
>     digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>     > eku: id-kp-serverAuth,id-kp-clientAuth
>     > pre-save command:
>     > post-save command: /usr/lib64/ipa/certmonger/restart_httpd
>     > track: yes
>     > auto-renew: yes
>
>     Since you have Apache up and limping along I'd first try:
>
>     # getcert resubmit -i 20150316184529
>
>     It is very possible that the tool will reject the server cert since it
>     is expired. If that happens then you'll need to set time back.
>
>     I'd encourage you to run: getcert list | grep expires
>
>     This will show you what certs need to be renewed and should give some
>     insight into the window of opportunity for setting the date back.
>
>     Assuming it's just the Apache cert that is expired I'd try setting the
>     date back to March 15, restart IPA, then I'd restart certmonger or try
>     the about resubmit command again.
>
>     Things can get tricky when some certs have renewed and some haven't.
>
>     rob
>
>     >
>     >
>     > On Wed, 7 Jun 2017 at 15:36 Roberto Cornacchia
>     > <roberto.cornacchia@gmail.com
>     <mailto:roberto.cornacchia@gmail.com>
>     <mailto:roberto.cornacchia@gmail.com
>     <mailto:roberto.cornacchia@gmail.com>>> wrote:
>     >
>     >     Thanks Rob,
>     >
>     >     I've seen in similar posts that you recommend to go back in
>     time and
>     >     restart certmonger. Would it work in this case?
>     >
>     >
>     >     On Wed, 7 Jun 2017 at 15:30 Rob Crittenden
>     <rcritten@redhat.com <mailto:rcritten@redhat.com>
>     >     <mailto:rcritten@redhat.com <mailto:rcritten@redhat.com>>> wrote:
>     >
>     >         Roberto Cornacchia via FreeIPA-users wrote:
>     >         > OK, I did so and httpd restarts.
>     >         >
>     >         > $ openssl s_client -connect 127.0.0.1:443
>     <http://127.0.0.1:443>
>     >         <http://127.0.0.1:443> <http://127.0.0.1:443> -showcerts
>     >         > CONNECTED(00000003)
>     >         > depth=1 O = HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
>     <http://HQ.SPINQUE.COM>
>     >         <http://HQ.SPINQUE.COM>, CN = Certificate
>     >         > Authority
>     >         > verify return:1
>     >         > depth=0 O = HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
>     <http://HQ.SPINQUE.COM>
>     >         <http://HQ.SPINQUE.COM>, CN =
>     >         > spinque04.hq.spinque.com
>     <http://spinque04.hq.spinque.com> <http://spinque04.hq.spinque.com>
>     >         <http://spinque04.hq.spinque.com>
>     >         > verify error:num=10:certificate has expired
>     >         > notAfter=Mar 16 18:45:29 2017 GMT
>     >         > verify return:1
>     >         > depth=0 O = HQ.SPINQUE.COM <http://HQ.SPINQUE.COM>
>     <http://HQ.SPINQUE.COM>
>     >         <http://HQ.SPINQUE.COM>, CN =
>     >         > spinque04.hq.spinque.com
>     <http://spinque04.hq.spinque.com> <http://spinque04.hq.spinque.com>
>     >         <http://spinque04.hq.spinque.com>
>     >         > notAfter=Mar 16 18:45:29 2017 GMT
>     >         > verify return:1
>     >         > ---
>     >         > Certificate chain
>     >         >  0 s:/O=HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com
>     <http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
>     >         <http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
>     >         > <http://HQ.SPINQUE.COM/CN=spinque04.hq.spinque.com>
>     >         >    i:/O=HQ.SPINQUE.COM/CN=Certificate
>     <http://HQ.SPINQUE.COM/CN=Certificate>
>     >         <http://HQ.SPINQUE.COM/CN=Certificate>
>     >         > <http://HQ.SPINQUE.COM/CN=Certificate> Authority
>     >         > ...
>     >         >
>     >         > Fair enough, but why does this say it expires in 2019? Are
>     >         they two
>     >         > different certificates?
>     >         >
>     >         > $ getcert list -d /etc/httpd/alias -n ipaCert
>     >         > Number of certificates and requests being tracked: 8.
>     >         > Request ID '20160501114633':
>     >         > status: MONITORING
>     >         > stuck: no
>     >         > key pair storage:
>     >         >
>     >
>      type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>     >         > Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>     >         > certificate:
>     >         >
>     >
>      type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS
>     >         > Certificate DB'
>     >         > CA: dogtag-ipa-ca-renew-agent
>     >         > issuer: CN=Certificate Authority,O=HQ.SPINQUE.COM
>     <http://HQ.SPINQUE.COM>
>     >         <http://HQ.SPINQUE.COM> <http://HQ.SPINQUE.COM>
>     >         > subject: CN=IPA RA,O=HQ.SPINQUE.COM
>     <http://HQ.SPINQUE.COM> <http://HQ.SPINQUE.COM>
>     >         <http://HQ.SPINQUE.COM>
>     >         > expires: 2019-01-26 19:41:51 UTC
>     >         > key usage:
>     >
>      digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>     >         > eku: id-kp-serverAuth,id-kp-clientAuth
>     >         > pre-save command:
>     /usr/lib64/ipa/certmonger/renew_ra_cert_pre
>     >         > post-save command: /usr/lib64/ipa/certmonger/renew_ra_cert
>     >         > track: yes
>     >         > auto-renew: yes
>     >         >
>     >         > What's the right way to solve this?
>     >
>     >         You're looking at the wrong cert.
>     >
>     >         # getcert list -d /etc/httpd/alias -n Server-Cert
>     >
>     >         And really, you should examine all certificate status, not
>     just
>     >         a single
>     >         one.
>     >
>     >         I was also strongly urge you to wait until all problems
>     are resolved
>     >         before attempting to update packages in the future (unless
>     a package
>     >         claims to fix a specific problem), particularly when it
>     comes to
>     >         certificates.
>     >
>     >         rob
>     >
>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
>