I have not been able to renew the expired certificates yet. I would
appreciate help if possible.
Followup summary:
Q: Seems like part of the problem is that the KDC was not running. Had
you
done ipactl stop prior to the upgrade?
A: I could not get the KDC to stay running. So yes it was off during
the upgrade.
Q: Did it end up creating the tracking? Are there expired certs?
A: I was able to get the upgrade to finish successfully, after
restoring the server from VM snapshot, rolling back the system date,
and trying the update again. It did create the cert tracking!!! Yes
there are expired certs.
Q: As an aside, I'd have suggest deferring the package upgrade until
after
the other things were sorted. It just adds another moving part. Water
under the bridge now.
A: Yes sorry.
> On 2/5/2019 11:18 AM, Rob Crittenden wrote:
>> Chris Mohler wrote:
>>> Well... That was a mess.
>>>
>>> The ipa-server-upgrade didn't go so well. It failed and now my
>>> ca-replication master is broken. Here are the details. Any hope?
>>>
>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>>>> [1/11]: stopping directory server
>>>> [2/11]: saving configuration
>>>> [3/11]: disabling listeners
>>>> [4/11]: enabling DS global lock
>>>> [5/11]: disabling Schema Compat
>>>> [6/11]: starting directory server
>>>> [7/11]: updating schema
>>>> [8/11]: upgrading server
>>>> [9/11]: stopping directory server
>>>> [10/11]: restoring configuration
>>>> [11/11]: starting directory server
>>>> Done.
>>>> Update complete
>>>> Upgrading IPA services
>>>> Upgrading the configuration of the IPA services
>>>> [Verifying that root certificate is published]
>>>> [Migrate CRL publish directory]
>>>> CRL tree already moved
>>>> [Verifying that CA proxy configuration is correct]
>>>> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
>>>> command ipa-server-upgrade manually.
>>>> CA did not start in 300.0s
>>>> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
>>>> for
>>>> more information
>> Seems like part of the problem is that the KDC was not running. Had you
>> done ipactl stop prior to the upgrade?
>>
>> Did it end up creating the tracking? Are there expired certs?
>>
>> As an aside, I'd have suggest deferring the package upgrade until after
>> the other things were sorted. It just adds another moving part. Water
>> under the bridge now.
>>
>> rob
>>
>>> Here is a wall of errors from my /var/log/ipaupgrade.log
>>>> Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.947136504 -0500]
>>>> - ERR - set_krb5_creds - Could not get initial credentials for
>>>> principal [ldap/ipa2.domain.com(a)domain.com] in keytab
>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
>>>> requested realm)
>>>> Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.953577522 -0500]
>>>> - ERR - slapi_ldap_bind - Error: could not send startTLS request:
>>>> error -1 (Can't contact LDAP server) errno 107 (Transport endpoint
is
>>>> not connected)
>>>> Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.958062514 -0500]
>>>> - ERR - set_krb5_creds - Could not get initial credentials for
>>>> principal [ldap/ipa2.domain.com(a)domain.com] in keytab
>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC for
>>>> requested realm)
>>>> Feb 4 17:47:33 ipa2 ns-slapd: GSSAPI Error: Unspecified GSS failure.
>>>> Minor code may provide more information (No Kerberos credentials
>>>> available (default cache: /tmp/krb5cc_389))
>>>> Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.965496432 -0500]
>>>> - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication
>>>> Manager
>>>> masterAgreement1-ipa2.domain.com-pki-tomcat,ou=csusers,cn=config]
>>>> authentication mechanism [SIMPLE]: error 32 (No such object)
>>>> Feb 4 17:47:40 ipa2 server: WARNING: Exception processing realm
>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>> Feb 4 17:47:40 ipa2 server: javax.ws.rs.ServiceUnavailableException:
>>>> Subsystem unavailable
>>>> Feb 4 17:47:40 ipa2 server: at
>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>
>>>> Feb 4 17:47:40 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>
>>>>
>>>> Feb 4 17:47:40 ipa2 server: at
>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>
>>>>
>>>> Feb 4 17:47:40 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>
>>>>
>>>> Feb 4 17:47:40 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:47:40 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:47:40 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>
>>>>
>>>> Feb 4 17:47:40 ipa2 server: at java.lang.Thread.run(Thread.java:748)
>>>> Feb 4 17:47:41 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>> Feb 4 17:47:50 ipa2 server: WARNING: Exception processing realm
>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>> Feb 4 17:47:50 ipa2 server: javax.ws.rs.ServiceUnavailableException:
>>>> Subsystem unavailable
>>>> Feb 4 17:47:50 ipa2 server: at
>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>
>>>> Feb 4 17:47:50 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>
>>>>
>>>> Feb 4 17:47:50 ipa2 server: at
>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>
>>>>
>>>> Feb 4 17:47:50 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>
>>>>
>>>> Feb 4 17:47:50 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:47:50 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:47:50 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>
>>>>
>>>> Feb 4 17:47:50 ipa2 server: at java.lang.Thread.run(Thread.java:748)
>>>> Feb 4 17:47:52 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>> Feb 4 17:48:00 ipa2 server: WARNING: Exception processing realm
>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>> Feb 4 17:48:00 ipa2 server: javax.ws.rs.ServiceUnavailableException:
>>>> Subsystem unavailable
>>>> Feb 4 17:48:00 ipa2 server: at
>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>
>>>> Feb 4 17:48:00 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>
>>>>
>>>> Feb 4 17:48:00 ipa2 server: at
>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>
>>>>
>>>> Feb 4 17:48:00 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>
>>>>
>>>> Feb 4 17:48:00 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:48:00 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:48:00 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>
>>>>
>>>> Feb 4 17:48:00 ipa2 server: at java.lang.Thread.run(Thread.java:748)
>>>> Feb 4 17:48:02 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>> Feb 4 17:48:10 ipa2 server: WARNING: Exception processing realm
>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>> Feb 4 17:48:10 ipa2 server: javax.ws.rs.ServiceUnavailableException:
>>>> Subsystem unavailable
>>>> Feb 4 17:48:10 ipa2 server: at
>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>
>>>> Feb 4 17:48:10 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>
>>>>
>>>> Feb 4 17:48:10 ipa2 server: at
>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>
>>>>
>>>> Feb 4 17:48:10 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>
>>>>
>>>> Feb 4 17:48:10 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:48:10 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:48:10 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>
>>>>
>>>> Feb 4 17:48:10 ipa2 server: at java.lang.Thread.run(Thread.java:748)
>>>> Feb 4 17:48:12 ipa2 [sssd[ldap_child[2284]]]: Failed to initialize
>>>> credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact
>>>> any
>>>> KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted
LDAP
>>>> connection.
>>>> Feb 4 17:48:12 ipa2 [sssd[ldap_child[2285]]]: Failed to initialize
>>>> credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact
>>>> any
>>>> KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted
LDAP
>>>> connection.
>>>> Feb 4 17:48:20 ipa2 server: WARNING: Exception processing realm
>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>> Feb 4 17:48:20 ipa2 server: javax.ws.rs.ServiceUnavailableException:
>>>> Subsystem unavailable
>>>> Feb 4 17:48:20 ipa2 server: at
>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>
>>>> Feb 4 17:48:20 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>
>>>>
>>>> Feb 4 17:48:20 ipa2 server: at
>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>
>>>>
>>>> Feb 4 17:48:20 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>
>>>>
>>>> Feb 4 17:48:20 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:48:20 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:48:20 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>
>>>>
>>>> Feb 4 17:48:20 ipa2 server: at java.lang.Thread.run(Thread.java:748)
>>>> Feb 4 17:48:22 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>> Feb 4 17:48:30 ipa2 server: WARNING: Exception processing realm
>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>> Feb 4 17:48:30 ipa2 server: javax.ws.rs.ServiceUnavailableException:
>>>> Subsystem unavailable
>>>> Feb 4 17:48:30 ipa2 server: at
>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>
>>>> Feb 4 17:48:30 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>
>>>>
>>>> Feb 4 17:48:30 ipa2 server: at
>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>
>>>>
>>>> Feb 4 17:48:30 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>
>>>>
>>>> Feb 4 17:48:30 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:48:30 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>>
>>>> Feb 4 17:48:30 ipa2 server: at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>
>>>>
>>>> Feb 4 17:48:30 ipa2 server: at java.lang.Thread.run(Thread.java:748)
>>>> Feb 4 17:48:31 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>> ^C
>>>> [root@ipa2 log]# less /var/log/ipaupgrade.log
>>>> <p><b>note</b> <u>The full stack trace of the
root cause is available
>>>> in the Apache Tomcat/7.0.76 logs.</u></p><HR
size="1"
>>>> noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
>>>> 2019-02-04T22:46:13Z DEBUG The CA status is: check interrupted due to
>>>> error: Retrieving CA status failed with status 500
>>>> 2019-02-04T22:46:13Z DEBUG Waiting for CA to start...
>>>> 2019-02-04T22:46:14Z DEBUG request POST
>>>>
http://ipa2.domain.com:8080/ca/admin/ca/getStatus
>>>> 2019-02-04T22:46:14Z DEBUG request body ''
>>>> 2019-02-04T22:46:14Z DEBUG response status 500
>>>> 2019-02-04T22:46:14Z DEBUG response headers Server: Apache-Coyote/1.1
>>>> Content-Type: text/html;charset=utf-8
>>>> Content-Language: en
>>>> Content-Length: 2208
>>>> Date: Mon, 04 Feb 2019 22:46:14 GMT
>>>> Connection: close
>>>>
>>>> 2019-02-04T22:46:14Z DEBUG response body
'<html><head><title>Apache
>>>> Tomcat/7.0.76 - Error report</title><style><!--H1
>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
>>>>
>>>> H2
>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
>>>>
>>>> H3
>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
>>>>
>>>> BODY
>>>> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
>>>>
>>>> B
>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
>>>>
>>>> P
>>>>
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
>>>>
>>>> {color : black;}A.name {color : black;}HR {color :
>>>> #525D76;}--></style> </head><body><h1>HTTP
Status 500 - Subsystem
>>>> unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b>
>>>> Exception report</p><p><b>message</b>
<u>Subsystem
>>>> unavailable</u></p><p><b>description</b>
<u>The server encountered an
>>>> internal error that prevented it from fulfilling this
>>>> request.</u></p><p><b>exception</b>
>>>> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem
>>>>
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
>>>>
>>>> <u>The full stack trace of the root cause is available in the
Apache
>>>> Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
>>>> Tomcat/7.0.76</h3></body></html>'
>>>> 2019-02-04T22:46:14Z DEBUG The CA status is: check interrupted due to
>>>> error: Retrieving CA status failed with status 500
>>>> 2019-02-04T22:46:14Z DEBUG Waiting for CA to start...
>>>> 2019-02-04T22:46:15Z DEBUG request POST
>>>>
http://ipa2.domain.com:8080/ca/admin/ca/getStatus
>>>> 2019-02-04T22:46:15Z DEBUG request body ''
>>>> 2019-02-04T22:46:15Z DEBUG response status 500
>>>> 2019-02-04T22:46:15Z DEBUG response headers Server: Apache-Coyote/1.1
>>>> Content-Type: text/html;charset=utf-8
>>>> Content-Language: en
>>>> Content-Length: 2208
>>>> Date: Mon, 04 Feb 2019 22:46:15 GMT
>>>> Connection: close
>>>>
>>>> 2019-02-04T22:46:15Z DEBUG response body
'<html><head><title>Apache
>>>> Tomcat/7.0.76 - Error report</title><style><!--H1
>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
>>>>
>>>> H2
>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
>>>>
>>>> H3
>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
>>>>
>>>> BODY
>>>> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
>>>>
>>>> B
>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
>>>>
>>>> P
>>>>
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
>>>>
>>>> {color : black;}A.name {color : black;}HR {color :
>>>> #525D76;}--></style> </head><body><h1>HTTP
Status 500 - Subsystem
>>>> unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b>
>>>> Exception report</p><p><b>message</b>
<u>Subsystem
>>>> unavailable</u></p><p><b>description</b>
<u>The server encountered an
>>>> internal error that prevented it from fulfilling this
>>>> request.</u></p><p><b>exception</b>
>>>> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem
>>>>
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
>>>>
>>>> <u>The full stack trace of the root cause is available in the
Apache
>>>> Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
>>>> Tomcat/7.0.76</h3></body></html>'
>>>> 2019-02-04T22:46:15Z DEBUG The CA status is: check interrupted due to
>>>> error: Retrieving CA status failed with status 500
>>>> 2019-02-04T22:46:15Z DEBUG Waiting for CA to start...
>>>> 2019-02-04T22:46:16Z ERROR IPA server upgrade failed: Inspect
>>>> /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
>>>> 2019-02-04T22:46:16Z DEBUG File
>>>> "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line
178,
>>>> in execute
>>>> return_value = self.run()
>>>> File
>>>>
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
>>>>
>>>> line 56, in run
>>>> raise admintool.ScriptError(str(e))
>>>>
>>>> 2019-02-04T22:46:16Z DEBUG The ipa-server-upgrade command failed,
>>>> exception: ScriptError: CA did not start in 300.0s
>>>> 2019-02-04T22:46:16Z ERROR CA did not start in 300.0s
>>>> 2019-02-04T22:46:16Z ERROR The ipa-server-upgrade command failed. See
>>>> /var/log/ipaupgrade.log for more information
>>> Thanks,
>>> -Chris
>>>
>>>> Rob,
>>>>
>>>> I'll be honest. I think you are suggesting an ldapsearch with this
>>>>
>>>> Check to see which masteris the renewal master. Look in
>>>> cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
>>>> ipaConfigString=caRenewalMaster
>>>>
>>>> sorry I've not figured out how to successfully ldapsearch
>>>>
>>>> Instead I did this:
>>>> ipa config-show |grep 'CA renewal master'
>>>>
>>>> It came up blank. I suspect I didn't have a renewal master somehow.
>>>>
>>>> Then I did This:
>>>> ipa-csreplica-manage set-renewal-master ipa2 (hostname of working IPA
>>>> server)
>>>>
>>>> Next is a "yum update" to be safe, and lastly
"ipa-server-upgrade" on
>>>> ipa2. When that's all done I'll try "yum update" and
>>>> "ipa-server-upgrade" on my broken IPA system ipa1
>>>>
>>>> I'll report back here when finished.
>>>>
>>>> Thanks,
>>>>
>>>> -Chris
>>>>
>>>>> Check to see which masteris the renewal master. Look in
>>>>> cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
>>>>> ipaConfigString=caRenewalMaster
>>>>>
>>>>> You want to run the script on that master first to get the certs
>>>>> renewed.
>>>>>
>>>>> I'd start by re-running ipa-server-upgrade. It is idempotent so
>>>>> there
>>>>> should be no risk. It may repair the tracking for you.
>>>>>
>>>>> rob
>>>> On 2/4/2019 3:30 PM, Rob Crittenden wrote:
>>>>> Chris Mohler via FreeIPA-users wrote:
>>>>>> Thanks for looking at my issue!
>>>>>>
>>>>>> There have been no recent updates on my system. Actually I was
>>>>>> getting
>>>>>> ready to update when I noticed things weren't good.
>>>>>>
>>>>>> Here is the output from the log of the most recent update. Looks
>>>>>> like it
>>>>>> was completed successfully. The lines you asked about are in
>>>>>> Bold/underlined.
>>>>>>
>>>>>>> 2018-07-18T16:55:21Z INFO [Update certmonger certificate
renewal
>>>>>>> configuration]
>>>>>>> 2018-07-18T16:55:21Z DEBUG Loading Index file from
>>>>>>> '/var/lib/ipa/sysrestore/sysrestore.index'
>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/usr/bin/certutil -d
>>>>>>> /etc/pki/pki-tomcat/alias -L -f
>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt
>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0
>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=
>>>>>>> Certificate Nickname Trust
>>>>>>> Attributes
>>>>>>> SSL,S/MIME,JAR/XPI
>>>>>>>
>>>>>>> caSigningCert cert-pki-ca CTu,Cu,Cu
>>>>>>> subsystemCert cert-pki-ca u,u,u
>>>>>>> ocspSigningCert cert-pki-ca u,u,u
>>>>>>> auditSigningCert cert-pki-ca u,u,Pu
>>>>>>> Server-Cert cert-pki-ca u,u,u
>>>>>>>
>>>>>>> 2018-07-18T16:55:21Z DEBUG stderr=
>>>>>>> _*2018-07-18T16:55:21Z DEBUG Configuring certmonger to stop
>>>>>>> tracking
>>>>>>> system certificates for CA*_
>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start
>>>>>>> messagebus.service
>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0
>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=
>>>>>>> 2018-07-18T16:55:21Z DEBUG stderr=
>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active
>>>>>>> messagebus.service
>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0
>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=active
>>>>>>>
>>>>>>> 2018-07-18T16:55:21Z DEBUG stderr=
>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start
>>>>>>> certmonger.service
>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0
>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=
>>>>>>> 2018-07-18T16:55:21Z DEBUG stderr=
>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl is-active
>>>>>>> certmonger.service
>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return code=0
>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=active
>>>>>>>
>>>>>> -snip- a few more lines like the section above.
>>>>>>> 2018-07-18T16:55:25Z DEBUG stderr=
>>>>>>> 2018-07-18T16:55:30Z DEBUG Loading Index file from
>>>>>>> '/var/lib/ipa/sysrestore/sysrestore.index'
>>>>>>> 2018-07-18T16:55:30Z DEBUG Starting external process
>>>>>>> 2018-07-18T16:55:30Z DEBUG args=/usr/bin/certutil -d
>>>>>>> /etc/dirsrv/DOMAINNAMEHERE -L -n Server-Cert -a -f
>>>>>>> /etc/dirsrv/DOMAINNAMEHERE/pwdfile.txt
>>>>>>> 2018-07-18T16:55:30Z DEBUG Process finished, return code=0
>>>>>>> 2018-07-18T16:55:30Z DEBUG stdout=-----BEGIN
CERTIFICATE-----
>>>>>> -Snip- Cert and Key stuff goes here-
>>>>>>> 2018-07-18T16:55:34Z DEBUG stderr=
>>>>>>> _*2018-07-18T16:55:35Z INFO Certmonger certificate renewal
>>>>>>> configuration updated*_
>>>>> Check to see which masteris the renewal master. Look in
>>>>> cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
>>>>> ipaConfigString=caRenewalMaster
>>>>>
>>>>> You want to run the script on that master first to get the certs
>>>>> renewed.
>>>>>
>>>>> I'd start by re-running ipa-server-upgrade. It is idempotent so
>>>>> there
>>>>> should be no risk. It may repair the tracking for you.
>>>>>
>>>>> rob
>>>>>
>>>>>> On 2/4/2019 1:44 PM, Florence Blanc-Renaud wrote:
>>>>>>> On 2/4/19 5:59 PM, Chris Mohler via FreeIPA-users wrote:
>>>>>>>> Hi Everyone,
>>>>>>>>
>>>>>>>> I'm looking for some help. I'm having trouble
with everything
>>>>>>>> basically.
>>>>>>>>
>>>>>>>> I think one of my CA's certs expired or something. I
can't kinit
>>>>>>>> admin, I can't login via the WebGui. If I
"getcert list" it
>>>>>>>> returns
>>>>>>>> "Number of certificates and requests being tracked:
0."
>>>>>>>>
>>>>>>>> This all started happening a few days ago and I am at a
loss
>>>>>>>> as to
>>>>>>>> what happened. On a whim I set the system date and time
back a
>>>>>>>> few
>>>>>>>> months to see if my certs were expired and like magic I
can
>>>>>>>> login to
>>>>>>>> the Webgui but I'm still not tracking anything with
"getcert
>>>>>>>> list" I
>>>>>>>> suspect the cert has expired but without tracking it I
can't
>>>>>>>> tell, or
>>>>>>>> renew it.
>>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> can you check if an upgrade happened recently (have a look
at
>>>>>>> /var/log/ipaupgrade.log)? The upgrade stop tracking certs
and
>>>>>>> re-configures certmonger, so if it failed in the middle you
may be
>>>>>>> left without any tracking.
>>>>>>> You should be able to find lines like the following if the
>>>>>>> untracking/tracking went fine:
>>>>>>> ---
>>>>>>> [Update certmonger certificate renewal configuration]
>>>>>>> Configuring certmonger to stop tracking system certificates
for CA
>>>>>>> Certmonger certificate renewal configuration updated
>>>>>>> ---
>>>>>>>
>>>>>>> HTH,
>>>>>>> flo
>>>>>>>
>>>>>>>> Please help
>>>>>>>>
>>>>>>>> I'm running Centos 7, FreeIPA 4.5.4
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> -Chris
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> FreeIPA-users mailing list --
>>>>>>>> freeipa-users(a)lists.fedorahosted.org
>>>>>>>> To unsubscribe send an email to
>>>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>>> Fedora Code of Conduct:
>>>>>>>>
https://getfedora.org/code-of-conduct.html
>>>>>>>> List Guidelines:
>>>>>>>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>>>> List Archives:
>>>>>>>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>> To unsubscribe send an email to
>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>>> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
>>>>>> List Guidelines:
>>>>>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>> List Archives:
>>>>>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>
>>>>>>
>>>>>>