Solved!
I had orphaned RUV's...
I made sure I was logged into the server that is the CA replication master:
#ipa config-show | grep 'CA renewal master'
I listed all my replicas:
#ipa-replica-manage list
I removed all replicas (other than the box I am logged into)
#"ipa-replica-manage del"
I listed all the ruv's:
#ipa-replica-manage list-ruv
I removed all the ruv's
# ipa-replica-manage clean-ruv
Then:
#service ntpd stop
after doing that I set the date back to when my certs were still valid.
#date 122910262018
Start all the IPA services manually and watch for errors:
#systemctl restart dirsrv(a)example.com
#systemctl restart krb5kdc
#systemctl restart httpd
#systemctl start kadmin
#systemctl start ipa-custodia
#systemctl start pki-tomcatd(a)pki-tomcat.service
##never figured out the one for the ipa-otpd Service (good luck)
Magically they all started without the replicas and ruv's!!
#getcert list
Confirms my two certs that are about to expire
Try to renew them with a:
#service certmonger restart
Got the auditSigningCert to renew automatically with the certmonger
restart. The ocspSigningCert was more stubborn:
#getcert resubmit -i 20181230160146
did the trick!
#getcert list
confirms all my dates are good.
#ntpdate
time.yourserver.com
#ipactl restart
all was better! Hope this helps someone, maybe a future me.
Alas I still have one question... How bad is "unable to decode: {replica
3}" below and how do I fix it?
[root@ipa1 log]# ipa-replica-manage list-ruv
Directory Manager password:
unable to decode: {replica 3} 54e37389000100030000 54e37389000100030000
Replica Update Vectors:
ipa1.cs.oberlin.edu:389: 10
Certificate Server Replica Update Vectors:
ipa1.cs.oberlin.edu:389: 1095
ipa1.cs.oberlin.edu:389: 96