Rob Foehl wrote:
On Thu, 15 Jun 2017, Rob Crittenden wrote:
> Rob Foehl wrote:
>> Can I at least get a yes or no on whether external CA certificate
>> renewal has ever been tested when that certificate is nearing
>> expiration?
>
> Yes. I tested this with IPA v3.0. Did it break in between? Possible.
>
> As I pointed out certmonger is unaware of the certificate chain and
> focuses only on the cert not-after date and resubmits the CSR to the CA
> that issued the certificate originally.
Thanks for the reply.
certmonger not knowing about the chain is understandable, as is the
resubmission of each tracked cert to the existing CA. Doing this
results in a pile of certs that expire relatively quickly, being tied to
the old CA, but that's also not surprising -- the surprise is that it
only did that once, and has since appeared to ignore them all, even
after the CA was renewed manually and the newly-issued-but-short-lived
certs tied to the old CA expired.
Ok, I'll need to try to reproduce it. It may take me a while to get
around to this so feel free to nag me.
>> I just duplicated last week's result using an earlier
snapshot of the
>> same VM and a renewed CA cert with a 3-day validity. certmonger ignored
>> every other cert that it already renewed once with the original CA;
>> whole system is hosed after the original cert expires. It's probably
>> possible to recover by manually replacing every certificate, but I
>> haven't had time to try that.
>
> certmonger checks at days 28, 7, 3, 2 and 1 before expiration by default
> for certificate expiration so it should have looked at the certs at
> least two times, three depending on timing (and really, it's seconds
> before expiration). Did you let the system sit for 3 days before things
> died? Was anything logged to syslog? Moving time forward a day at a time
> is insufficient to test this without restarting certmonger.
I let the original VM snapshot run for a month straight, renewing the
IPA CA by hand after the first round of certmonger-initiated renewals
with 14 days til expiration and on the second attempt after expiration.
The first attempt used another 30-day cert, the second used a 3-day and
was allowed to run straight through. No time jumps while the VM is
running, and all snapshots with the VM powered off, so it always booted
with an accurate clock.
certmonger never logged anything after the first renewal cycle on either
attempt. A 'getcert list' on the long-running VM shows all of the
tracked certificates with an expiration date of 2017-06-24, which
matches the lifetime of the renewed CA cert, but none of the services
attempting to load or use them are happy.
It depends on why they aren't happy. Are they not happy due to expired
certs or something else?
[snip]
But httpd still refuses to start with that NSSDB, and this appears to be
why:
# certutil -L -n Signing-Cert -d /etc/httpd/alias
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 9 (0x9)
Signature Algorithm: PKCS #1 SHA-256 With RSA Encryption
Issuer: "CN=Certificate Authority,O=EXAMPLE.COM"
Validity:
Not Before: Mon May 08 06:33:16 2017
Not After : Wed Jun 07 06:25:53 2017
Subject: "CN=Object Signing Cert,O=EXAMPLE.COM"
mod_nss shouldn't be considering the signing cert so I doubt this is
related.
Does certmonger know how to replace the entire certificate chain in the
respective store(s)?
(The third certificate in there, ipaCert / CN=IPA RA, has the same dates
as the Server-Cert above.)
So it was renewed as well.
certmonger doesn't push out new chains so if that changed in between
that would do it. This is another way to test cert validation from the
command-line:
# certutil -V -u V -d /etc/httpd/alias -n Server-Cert
If you want to see if updating the CA cert(s) makes any difference.
> Even in a worst-case scenario, where all the certs expire, it is a
> fairly straightforward process to get the services back up by going back
> in time, renewing the IPA CA then restarting certmonger to renew the
> service certificates.
>
> Is it perfect? No. A search of the users forum should make that
> apparent. It has been difficult to reproduce the failures because it's
> difficult to simulate by moving time around. Several years ago I left
> VMs running for months to try to simulate failures and it always worked
> for me.
I haven't tried kicking the clock around yet... The second attempt
booted from a month-old snapshot and immediately blew itself up;
renewing the CA cert and restarting certmonger (really, the whole VM)
didn't change anything.
If the chain changes then yeah, that'd cause problems.
> Note too that there is a difference between certmonger and the
renewals.
> certmonger renews certs but there are helpers that need to fire off to
> update information within IPA as well and to distribute updated
> certificates to replicas. These scripts were updated significantly since
> I wrote them to be much more robust in terms of reliability and logging.
Consider uses of "certmonger" above to include these... Another
wrinkle, discovered early on, was broken SELinux policy that prevented
certmonger from running any of them. That was (apparently) fixed by a
later selinux-policy-targeted package release, but I haven't tried the
whole process from a bare install since. The second test with the 3-day
lifetime on the IPA CA renewal should've been okay here. I can try
again with a fresh install and relatively short IPA CA cert lifetimes,
say 4 days per renewal if that'll be sufficient to provoke this a bit
faster.
I'm still worried about the missing "phase 2" when it comes to
distributing a new external CA certificate -- the CA I have expires in 3
years, and it'd be nice to know whether I'm shooting myself in the foot
if I try signing the for-real IPA CA with it now.
The really tricky bit is distributing the updated CA chain around. I've
been away from IPA for a while but I can give you some bread crumbs. I
believe that ipa-cacert-manage can be used to update the stored CA chain
in LDAP and then running ipa-certupdate will pull the chain down, it
just needs to be run on every master and client.
The biggest gap here is any notification of impending doom: YOUR CA WILL
EXPIRE IN 'n' DAYS.
This gap is due to no real way of notifying anyone. We could nag in the
UI perhaps, or log, but that wouldn't guarantee that an admin would see it.
E-mail might work but it requires a configured MUA and that can't be
assumed, and I seriously doubt a typical IPA admin wants yet another
service to configure.
rob