Well that did it. Thanks!
So for people following this thread, what would have caused this? Were
they not set up correctly on installation, or were they somehow removed?
If the former, how long could the system run with the incorrect
configuration? If the latter, how likely is this to recur?
And yes, I’m glad the upstream error message is being improved :)
I'm glad it fixed it for you. Yes, the new message will say something
like "CA foo not found" or something similar.
I'm not entirely sure why this happens. It has been pretty rare so it
must be some odd corner case. The upgrade code tweaks the CA's in some
cases (depending on version) so I guess it's possible that's it, but it
may be the sort of thing where there is a lot of intervening time
between when it breaks and when it gets noticed.
I'm pretty certain it is not user-error but it hasn't happened enough,
and it's usually found at pretty catastrophic times where getting the
system back up has more priority than root cause, for us to find a cause
yet.
The ipa-healthcheck tool will look for this though so there is a way to
proactively tell whether it is an issue. Food for though anyway.
rob
> On Mar 5, 2020, at 2:51 PM, Rob Crittenden <rcritten(a)redhat.com
> <mailto:rcritten@redhat.com>> wrote:
>
> You are missing two certmonger CA helpers, dogtag-ipa-ca-renew-agent and
> dogtag-ipa-ca-renew-agent-reuse.
>
> Re-add them as root with:
>
> # getcert add-ca -c dogtag-ipa-ca-renew-agent -e
> /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit
> # getcert add-ca -c dogtag-ipa-ca-renew-agent-reuse -e
> /usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit --reuse-existing
>
> Confirm with getcert list that every tracked request has a CA value.
>
> That should do the trick, you can now run:
>
> # ipa-server-upgrade
>
> rob
>
> David Carter via FreeIPA-users wrote:
>> With some redactions:
>>
>> # getcert list
>> Number of certificates and requests being tracked: 9.
>> Request ID '20190815160425':
>> status: MONITORING
>> stuck: no
>> key pair storage: type=FILE,location='/var/lib/ipa/ra-agent.key'
>> certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=IPA RA,O=XXXXXXXXXXXXX
>> expires: 2021-08-04 16:04:27 UTC
>> key usage: digitalSignature,keyEncipherment,dataEncipherment
>> eku: id-kp-serverAuth,id-kp-clientAuth
>> pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
>> post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
>> track: yes
>> auto-renew: yes
>> Request ID '20190815160528':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
>> cert-pki-ca',token='NSS Certificate DB'
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=CA Audit,O=XXXXXXXXXXXXX
>> expires: 2021-08-04 16:03:26 UTC
>> key usage: digitalSignature,nonRepudiation
>> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>> "auditSigningCert cert-pki-ca"
>> track: yes
>> auto-renew: yes
>> Request ID '20190815160529':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>> cert-pki-ca',token='NSS
>> Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
>> cert-pki-ca',token='NSS
>> Certificate DB'
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=OCSP Subsystem,O=XXXXXXXXXXXXX
>> expires: 2021-08-04 16:03:24 UTC
>> eku: id-kp-OCSPSigning
>> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>> "ocspSigningCert cert-pki-ca"
>> track: yes
>> auto-renew: yes
>> Request ID '20190815160530':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
>> cert-pki-ca',token='NSS Certificate DB'
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=CA Subsystem,O=XXXXXXXXXXXXX
>> expires: 2021-08-04 16:03:25 UTC
>> key usage:
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> eku: id-kp-clientAuth
>> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>> "subsystemCert cert-pki-ca"
>> track: yes
>> auto-renew: yes
>> Request ID '20190815160531':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
>> cert-pki-ca',token='NSS Certificate DB'
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> expires: 2039-08-15 16:03:22 UTC
>> key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
>> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>> "caSigningCert cert-pki-ca"
>> track: yes
>> auto-renew: yes
>> Request ID '20190815160532':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>> cert-pki-ca',token='NSS Certificate DB',pin set
>> certificate:
>>
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
>> cert-pki-ca',token='NSS Certificate DB'
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=XXXXXXXXXXXXX,O=XXXXXXXXXXXXX
>> expires: 2021-08-04 16:03:24 UTC
>> dns: XXXXXXXXXXXXX
>> key usage: digitalSignature,keyEncipherment,dataEncipherment
>> eku: id-kp-serverAuth
>> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
>> "Server-Cert cert-pki-ca"
>> track: yes
>> auto-renew: yes
>> Request ID '20190815160610':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/dirsrv/slapd-XXXXXXXXXXXXX',nickname='Server-Cert',token='NSS
>> Certificate
DB',pinfile='/etc/dirsrv/slapd-XXXXXXXXXXXXX/pwdfile.txt'
>> certificate:
>>
type=NSSDB,location='/etc/dirsrv/slapd-XXXXXXXXXXXXX',nickname='Server-Cert',token='NSS
>> Certificate DB'
>> CA: IPA
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=XXXXXXXXXXXXX,O=XXXXXXXXXXXXX
>> expires: 2021-08-15 16:06:11 UTC
>> dns: XXXXXXXXXXXXX
>> principal name: ldap/XXXXXXXXXXXXX@XXXXXXXXXXXXX
>> key usage:
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> eku: id-kp-serverAuth,id-kp-clientAuth
>> pre-save command:
>> post-save command:
>> /usr/libexec/ipa/certmonger/restart_dirsrv XXXXXXXXXXXXX
>> track: yes
>> auto-renew: yes
>> Request ID '20190815160720':
>> status: MONITORING
>> stuck: no
>> key pair storage:
>>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>> Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>> certificate:
>>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
>> Certificate DB'
>> CA: IPA
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=XXXXXXXXXXXXX,O=XXXXXXXXXXXXX
>> expires: 2021-08-15 16:07:21 UTC
>> dns: XXXXXXXXXXXXX
>> principal name: HTTP/XXXXXXXXXXXXX@XXXXXXXXXXXXX
>> key usage:
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> eku: id-kp-serverAuth,id-kp-clientAuth
>> pre-save command:
>> post-save command: /usr/libexec/ipa/certmonger/restart_httpd
>> track: yes
>> auto-renew: yes
>> Request ID '20190815160739':
>> status: MONITORING
>> stuck: no
>> key pair storage: type=FILE,location='/var/kerberos/krb5kdc/kdc.key'
>> certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
>> CA: IPA
>> issuer: CN=Certificate Authority,O=XXXXXXXXXXXXX
>> subject: CN=XXXXXXXXXXXXX,O=XXXXXXXXXXXXX
>> expires: 2021-08-15 16:07:40 UTC
>> principal name: krbtgt/XXXXXXXXXXXXX@XXXXXXXXXXXXX
>> key usage:
>> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>> eku: id-kp-serverAuth,id-pkinit-KPKdc
>> pre-save command:
>> post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert
>> track: yes
>> auto-renew: yes
>>
>>
>>> On Mar 4, 2020, at 12:16 PM, Rob Crittenden <rcritten(a)redhat.com
>>> <mailto:rcritten@redhat.com>
>>> <mailto:rcritten@redhat.com>> wrote:
>>>
>>> David Carter via FreeIPA-users wrote:
>>>> Freeipa was running fine, then I guess it tried to update and now I
>>>> have an unusable system and it's impacting other systems. I've
seen
>>>> this error in a few online postings but no fix that I can apply.
>>>>
>>>> When I run it manually:
>>>>
>>>> [root@kdc1 log]# ipa-server-upgrade
>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>>>> [1/11]: stopping directory server
>>>> [2/11]: saving configuration
>>>> [3/11]: disabling listeners
>>>> [4/11]: enabling DS global lock
>>>> [5/11]: disabling Schema Compat
>>>> [6/11]: starting directory server
>>>> [7/11]: updating schema
>>>> [8/11]: upgrading server
>>>> [9/11]: stopping directory server
>>>> [10/11]: restoring configuration
>>>> [11/11]: starting directory server
>>>> Done.
>>>> Update complete
>>>> Upgrading IPA services
>>>> Upgrading the configuration of the IPA services
>>>> [Verifying that root certificate is published]
>>>> [Migrate CRL publish directory]
>>>> CRL tree already moved
>>>> [Verifying that CA proxy configuration is correct]
>>>> [Verifying that KDC configuration is using ipa-kdb backend]
>>>> [Fix DS schema file syntax]
>>>> Syntax already fixed
>>>> [Removing RA cert from DS NSS database]
>>>> RA cert already removed
>>>> [Enable sidgen and extdom plugins by default]
>>>> [Updating HTTPD service IPA configuration]
>>>> [Updating HTTPD service IPA WSGI configuration]
>>>> Nothing to do for configure_httpd_wsgi_conf
>>>> [Updating mod_nss protocol versions]
>>>> Protocol versions already updated
>>>> [Updating mod_nss cipher suite]
>>>> [Updating mod_nss enabling OCSP]
>>>> [Fixing trust flags in /etc/httpd/alias]
>>>> Trust flags already processed
>>>> [Moving HTTPD service keytab to gssproxy]
>>>> [Removing self-signed CA]
>>>> [Removing Dogtag 9 CA]
>>>> [Checking for deprecated KDC configuration files]
>>>> [Checking for deprecated backups of Samba configuration files]
>>>> [Add missing CA DNS records]
>>>> IPA CA DNS records already processed
>>>> [Removing deprecated DNS configuration options]
>>>> DNS is not configured
>>>> [Ensuring minimal number of connections]
>>>> DNS is not configured
>>>> [Updating GSSAPI configuration in DNS]
>>>> DNS is not configured
>>>> [Updating pid-file configuration in DNS]
>>>> DNS is not configured
>>>> DNS is not configured
>>>> DNS is not configured
>>>> DNS is not configured
>>>> DNS is not configured
>>>> DNS is not configured
>>>> DNS is not configured
>>>> DNS is not configured
>>>> [Upgrading CA schema]
>>>> CA schema update complete (no changes)
>>>> [Verifying that CA audit signing cert has 2 year validity]
>>>> [Update certmonger certificate renewal configuration]
>>>> Failed to get request: bus, object_path and dbus_interface must not
>>>> be None.
>>>> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
>>>> command ipa-server-upgrade manually.
>>>> bus, object_path and dbus_interface must not be None.
>>>> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
>>>> for more information
>>>> The tail of the log file (it is very long):
>>>>
>>>> 2020-03-03T03:51:44Z DEBUG stderr=
>>>> 2020-03-03T03:51:44Z DEBUG Loading Index file from
>>>> '/var/lib/ipa/sysrestore/sysrestore.index'
>>>> 2020-03-03T03:51:44Z DEBUG Starting external process
>>>> 2020-03-03T03:51:44Z DEBUG args=/usr/bin/certutil -d
>>>> dbm:/etc/pki/pki-tomcat/alias -L -f
>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt
>>>> 2020-03-03T03:51:44Z DEBUG Process finished, return code=0
>>>> 2020-03-03T03:51:44Z DEBUG stdout=
>>>> Certificate Nickname Trust
>>>> Attributes
>>>>
SSL,S/MIME,JAR/XPI
>>>>
>>>> caSigningCert cert-pki-ca CTu,Cu,Cu
>>>> auditSigningCert cert-pki-ca u,u,Pu
>>>> Server-Cert cert-pki-ca u,u,u
>>>> ocspSigningCert cert-pki-ca u,u,u
>>>> subsystemCert cert-pki-ca u,u,u
>>>>
>>>> 2020-03-03T03:51:44Z DEBUG stderr=
>>>> 2020-03-03T03:51:44Z ERROR Failed to get request: bus, object_path
>>>> and dbus_interface must not be None.
>>>> 2020-03-03T03:51:44Z ERROR IPA server upgrade failed: Inspect
>>>> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
>>>> 2020-03-03T03:51:44Z DEBUG File
>>>> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line
178,
>>>> in execute
>>>> return_value = self.run()
>>>> File
>>>>
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
>>>> line 56, in run
>>>> raise admintool.ScriptError(str(e))
>>>>
>>>> 2020-03-03T03:51:44Z DEBUG The ipa-server-upgrade command failed,
>>>> exception: ScriptError: bus, object_path and dbus_interface must not
>>>> be None.
>>>> 2020-03-03T03:51:44Z ERROR bus, object_path and dbus_interface must
>>>> not be None.
>>>> 2020-03-03T03:51:44Z ERROR The ipa-server-upgrade command failed. See
>>>> /var/log/ipaupgrade.log for more information
>>>> I can run the certutil command separately, so it doesn't seem to be
>>>> that. I have no idea what the issue is, where to look, or how I can
>>>> fix this. Suggestions?
>>>>
>>>> OS: CentOS 7
>>>> IPA version: VERSION: 4.6.5, API_VERSION: 2.231
>>>
>>> What is the output of getcert list-cas
>>>
>>> One or more is missing.
>>>
>>> BTW this terrible error message is fixed upstream and will appear in
>>> future versions of RHEL 8 with a much better description.
>>>
>>> rob
>>>
>>
>>
>>
>> _______________________________________________
>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> <mailto:freeipa-users@lists.fedorahosted.org>
>> To unsubscribe send an email
>> to freeipa-users-leave(a)lists.fedorahosted.org
>> <mailto:freeipa-users-leave@lists.fedorahosted.org>
>> Fedora Code of
>>
Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> List
Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
>> List
>>
Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@list...