I have not been able to renew the expired certificates yet. I would
appreciate help if possible.
You will need to start repairing the CA renewal master. Can you provide
the output of
$ getcert list
on that node?
Depending on the list of expired certs and their validity dates we will
be able to suggest the next steps. It usually involves stopping ntpd,
setting the date back to a time when all the certs are valid and letting
certmonger do its job.
Any error related to certmonger in the journal may also help understand
why the renewal failed.
HTH,
flo
> Followup summary:
>
> Q: Seems like part of the problem is that the KDC was not running. Had
> you
> done ipactl stop prior to the upgrade?
>
> A: I could not get the KDC to stay running. So yes it was off during
> the upgrade.
>
> Q: Did it end up creating the tracking? Are there expired certs?
>
> A: I was able to get the upgrade to finish successfully, after
> restoring the server from VM snapshot, rolling back the system date,
> and trying the update again. It did create the cert tracking!!! Yes
> there are expired certs.
>
> Q: As an aside, I'd have suggest deferring the package upgrade until
> after
> the other things were sorted. It just adds another moving part. Water
> under the bridge now.
>
> A: Yes sorry.
>
>
>> On 2/5/2019 11:18 AM, Rob Crittenden wrote:
>>> Chris Mohler wrote:
>>>> Well... That was a mess.
>>>>
>>>> The ipa-server-upgrade didn't go so well. It failed and now my
>>>> ca-replication master is broken. Here are the details. Any hope?
>>>>
>>>>> Upgrading IPA:. Estimated time: 1 minute 30 seconds
>>>>> [1/11]: stopping directory server
>>>>> [2/11]: saving configuration
>>>>> [3/11]: disabling listeners
>>>>> [4/11]: enabling DS global lock
>>>>> [5/11]: disabling Schema Compat
>>>>> [6/11]: starting directory server
>>>>> [7/11]: updating schema
>>>>> [8/11]: upgrading server
>>>>> [9/11]: stopping directory server
>>>>> [10/11]: restoring configuration
>>>>> [11/11]: starting directory server
>>>>> Done.
>>>>> Update complete
>>>>> Upgrading IPA services
>>>>> Upgrading the configuration of the IPA services
>>>>> [Verifying that root certificate is published]
>>>>> [Migrate CRL publish directory]
>>>>> CRL tree already moved
>>>>> [Verifying that CA proxy configuration is correct]
>>>>> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
>>>>> command ipa-server-upgrade manually.
>>>>> CA did not start in 300.0s
>>>>> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log
>>>>> for
>>>>> more information
>>> Seems like part of the problem is that the KDC was not running. Had you
>>> done ipactl stop prior to the upgrade?
>>>
>>> Did it end up creating the tracking? Are there expired certs?
>>>
>>> As an aside, I'd have suggest deferring the package upgrade until after
>>> the other things were sorted. It just adds another moving part. Water
>>> under the bridge now.
>>>
>>> rob
>>>
>>>> Here is a wall of errors from my /var/log/ipaupgrade.log
>>>>> Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.947136504
-0500]
>>>>> - ERR - set_krb5_creds - Could not get initial credentials for
>>>>> principal [ldap/ipa2.domain.com(a)domain.com] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for
>>>>> requested realm)
>>>>> Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.953577522
-0500]
>>>>> - ERR - slapi_ldap_bind - Error: could not send startTLS request:
>>>>> error -1 (Can't contact LDAP server) errno 107 (Transport
endpoint is
>>>>> not connected)
>>>>> Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.958062514
-0500]
>>>>> - ERR - set_krb5_creds - Could not get initial credentials for
>>>>> principal [ldap/ipa2.domain.com(a)domain.com] in keytab
>>>>> [FILE:/etc/dirsrv/ds.keytab]: -1765328228 (Cannot contact any KDC
for
>>>>> requested realm)
>>>>> Feb 4 17:47:33 ipa2 ns-slapd: GSSAPI Error: Unspecified GSS
failure.
>>>>> Minor code may provide more information (No Kerberos credentials
>>>>> available (default cache: /tmp/krb5cc_389))
>>>>> Feb 4 17:47:33 ipa2 ns-slapd: [04/Feb/2019:17:47:33.965496432
-0500]
>>>>> - ERR - slapi_ldap_bind - Error: could not bind id [cn=Replication
>>>>> Manager
>>>>> masterAgreement1-ipa2.domain.com-pki-tomcat,ou=csusers,cn=config]
>>>>> authentication mechanism [SIMPLE]: error 32 (No such object)
>>>>> Feb 4 17:47:40 ipa2 server: WARNING: Exception processing realm
>>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>>> Feb 4 17:47:40 ipa2 server:
javax.ws.rs.ServiceUnavailableException:
>>>>> Subsystem unavailable
>>>>> Feb 4 17:47:40 ipa2 server: at
>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>>
>>>>> Feb 4 17:47:40 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>>
>>>>>
>>>>> Feb 4 17:47:40 ipa2 server: at
>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>>
>>>>>
>>>>> Feb 4 17:47:40 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>>
>>>>>
>>>>> Feb 4 17:47:40 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:47:40 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:47:40 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>>
>>>>>
>>>>> Feb 4 17:47:40 ipa2 server: at
java.lang.Thread.run(Thread.java:748)
>>>>> Feb 4 17:47:41 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>>> Feb 4 17:47:50 ipa2 server: WARNING: Exception processing realm
>>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>>> Feb 4 17:47:50 ipa2 server:
javax.ws.rs.ServiceUnavailableException:
>>>>> Subsystem unavailable
>>>>> Feb 4 17:47:50 ipa2 server: at
>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>>
>>>>> Feb 4 17:47:50 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>>
>>>>>
>>>>> Feb 4 17:47:50 ipa2 server: at
>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>>
>>>>>
>>>>> Feb 4 17:47:50 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>>
>>>>>
>>>>> Feb 4 17:47:50 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:47:50 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:47:50 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>>
>>>>>
>>>>> Feb 4 17:47:50 ipa2 server: at
java.lang.Thread.run(Thread.java:748)
>>>>> Feb 4 17:47:52 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>>> Feb 4 17:48:00 ipa2 server: WARNING: Exception processing realm
>>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>>> Feb 4 17:48:00 ipa2 server:
javax.ws.rs.ServiceUnavailableException:
>>>>> Subsystem unavailable
>>>>> Feb 4 17:48:00 ipa2 server: at
>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>>
>>>>> Feb 4 17:48:00 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>>
>>>>>
>>>>> Feb 4 17:48:00 ipa2 server: at
>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>>
>>>>>
>>>>> Feb 4 17:48:00 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>>
>>>>>
>>>>> Feb 4 17:48:00 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:48:00 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:48:00 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>>
>>>>>
>>>>> Feb 4 17:48:00 ipa2 server: at
java.lang.Thread.run(Thread.java:748)
>>>>> Feb 4 17:48:02 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>>> Feb 4 17:48:10 ipa2 server: WARNING: Exception processing realm
>>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>>> Feb 4 17:48:10 ipa2 server:
javax.ws.rs.ServiceUnavailableException:
>>>>> Subsystem unavailable
>>>>> Feb 4 17:48:10 ipa2 server: at
>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>>
>>>>> Feb 4 17:48:10 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>>
>>>>>
>>>>> Feb 4 17:48:10 ipa2 server: at
>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>>
>>>>>
>>>>> Feb 4 17:48:10 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>>
>>>>>
>>>>> Feb 4 17:48:10 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:48:10 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:48:10 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>>
>>>>>
>>>>> Feb 4 17:48:10 ipa2 server: at
java.lang.Thread.run(Thread.java:748)
>>>>> Feb 4 17:48:12 ipa2 [sssd[ldap_child[2284]]]: Failed to initialize
>>>>> credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact
>>>>> any
>>>>> KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted
LDAP
>>>>> connection.
>>>>> Feb 4 17:48:12 ipa2 [sssd[ldap_child[2285]]]: Failed to initialize
>>>>> credentials using keytab [MEMORY:/etc/krb5.keytab]: Cannot contact
>>>>> any
>>>>> KDC for realm 'domain.com'. Unable to create GSSAPI-encrypted
LDAP
>>>>> connection.
>>>>> Feb 4 17:48:20 ipa2 server: WARNING: Exception processing realm
>>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>>> Feb 4 17:48:20 ipa2 server:
javax.ws.rs.ServiceUnavailableException:
>>>>> Subsystem unavailable
>>>>> Feb 4 17:48:20 ipa2 server: at
>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>>
>>>>> Feb 4 17:48:20 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>>
>>>>>
>>>>> Feb 4 17:48:20 ipa2 server: at
>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>>
>>>>>
>>>>> Feb 4 17:48:20 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>>
>>>>>
>>>>> Feb 4 17:48:20 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:48:20 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:48:20 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>>
>>>>>
>>>>> Feb 4 17:48:20 ipa2 server: at
java.lang.Thread.run(Thread.java:748)
>>>>> Feb 4 17:48:22 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>>> Feb 4 17:48:30 ipa2 server: WARNING: Exception processing realm
>>>>> com.netscape.cms.tomcat.ProxyRealm@3badc78b background process
>>>>> Feb 4 17:48:30 ipa2 server:
javax.ws.rs.ServiceUnavailableException:
>>>>> Subsystem unavailable
>>>>> Feb 4 17:48:30 ipa2 server: at
>>>>>
com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>>
>>>>> Feb 4 17:48:30 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>>
>>>>>
>>>>> Feb 4 17:48:30 ipa2 server: at
>>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>>
>>>>>
>>>>> Feb 4 17:48:30 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>>
>>>>>
>>>>> Feb 4 17:48:30 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:48:30 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>>
>>>>>
>>>>> Feb 4 17:48:30 ipa2 server: at
>>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>>
>>>>>
>>>>> Feb 4 17:48:30 ipa2 server: at
java.lang.Thread.run(Thread.java:748)
>>>>> Feb 4 17:48:31 ipa2 dhclient[598]: DHCPREQUEST on eth0 to
>>>>> 132.162.1.131 port 67 (xid=0x27e7db13)
>>>>> ^C
>>>>> [root@ipa2 log]# less /var/log/ipaupgrade.log
>>>>> <p><b>note</b> <u>The full stack trace of the
root cause is available
>>>>> in the Apache Tomcat/7.0.76 logs.</u></p><HR
size="1"
>>>>> noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
>>>>> 2019-02-04T22:46:13Z DEBUG The CA status is: check interrupted due
to
>>>>> error: Retrieving CA status failed with status 500
>>>>> 2019-02-04T22:46:13Z DEBUG Waiting for CA to start...
>>>>> 2019-02-04T22:46:14Z DEBUG request POST
>>>>>
http://ipa2.domain.com:8080/ca/admin/ca/getStatus
>>>>> 2019-02-04T22:46:14Z DEBUG request body ''
>>>>> 2019-02-04T22:46:14Z DEBUG response status 500
>>>>> 2019-02-04T22:46:14Z DEBUG response headers Server:
Apache-Coyote/1.1
>>>>> Content-Type: text/html;charset=utf-8
>>>>> Content-Language: en
>>>>> Content-Length: 2208
>>>>> Date: Mon, 04 Feb 2019 22:46:14 GMT
>>>>> Connection: close
>>>>>
>>>>> 2019-02-04T22:46:14Z DEBUG response body
'<html><head><title>Apache
>>>>> Tomcat/7.0.76 - Error report</title><style><!--H1
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
>>>>>
>>>>> H2
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
>>>>>
>>>>> H3
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
>>>>>
>>>>> BODY
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
>>>>>
>>>>> B
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
>>>>>
>>>>> P
>>>>>
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
>>>>>
>>>>> {color : black;}A.name {color : black;}HR {color :
>>>>> #525D76;}--></style> </head><body><h1>HTTP
Status 500 - Subsystem
>>>>> unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b>
>>>>> Exception report</p><p><b>message</b>
<u>Subsystem
>>>>>
unavailable</u></p><p><b>description</b> <u>The server
encountered an
>>>>> internal error that prevented it from fulfilling this
>>>>> request.</u></p><p><b>exception</b>
>>>>> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem
>>>>>
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
>>>>>
>>>>> <u>The full stack trace of the root cause is available in the
Apache
>>>>> Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
>>>>> Tomcat/7.0.76</h3></body></html>'
>>>>> 2019-02-04T22:46:14Z DEBUG The CA status is: check interrupted due
to
>>>>> error: Retrieving CA status failed with status 500
>>>>> 2019-02-04T22:46:14Z DEBUG Waiting for CA to start...
>>>>> 2019-02-04T22:46:15Z DEBUG request POST
>>>>>
http://ipa2.domain.com:8080/ca/admin/ca/getStatus
>>>>> 2019-02-04T22:46:15Z DEBUG request body ''
>>>>> 2019-02-04T22:46:15Z DEBUG response status 500
>>>>> 2019-02-04T22:46:15Z DEBUG response headers Server:
Apache-Coyote/1.1
>>>>> Content-Type: text/html;charset=utf-8
>>>>> Content-Language: en
>>>>> Content-Length: 2208
>>>>> Date: Mon, 04 Feb 2019 22:46:15 GMT
>>>>> Connection: close
>>>>>
>>>>> 2019-02-04T22:46:15Z DEBUG response body
'<html><head><title>Apache
>>>>> Tomcat/7.0.76 - Error report</title><style><!--H1
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
>>>>>
>>>>> H2
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
>>>>>
>>>>> H3
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
>>>>>
>>>>> BODY
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
>>>>>
>>>>> B
>>>>>
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
>>>>>
>>>>> P
>>>>>
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
>>>>>
>>>>> {color : black;}A.name {color : black;}HR {color :
>>>>> #525D76;}--></style> </head><body><h1>HTTP
Status 500 - Subsystem
>>>>> unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b>
>>>>> Exception report</p><p><b>message</b>
<u>Subsystem
>>>>>
unavailable</u></p><p><b>description</b> <u>The server
encountered an
>>>>> internal error that prevented it from fulfilling this
>>>>> request.</u></p><p><b>exception</b>
>>>>> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem
>>>>>
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre></p><p><b>note</b>
>>>>>
>>>>> <u>The full stack trace of the root cause is available in the
Apache
>>>>> Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
>>>>> Tomcat/7.0.76</h3></body></html>'
>>>>> 2019-02-04T22:46:15Z DEBUG The CA status is: check interrupted due
to
>>>>> error: Retrieving CA status failed with status 500
>>>>> 2019-02-04T22:46:15Z DEBUG Waiting for CA to start...
>>>>> 2019-02-04T22:46:16Z ERROR IPA server upgrade failed: Inspect
>>>>> /var/log/ipaupgrade.log and run command ipa-server-upgrade
manually.
>>>>> 2019-02-04T22:46:16Z DEBUG File
>>>>> "/usr/lib/python2.7/site-packages/ipapython/admintool.py",
line 178,
>>>>> in execute
>>>>> return_value = self.run()
>>>>> File
>>>>>
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
>>>>>
>>>>> line 56, in run
>>>>> raise admintool.ScriptError(str(e))
>>>>>
>>>>> 2019-02-04T22:46:16Z DEBUG The ipa-server-upgrade command failed,
>>>>> exception: ScriptError: CA did not start in 300.0s
>>>>> 2019-02-04T22:46:16Z ERROR CA did not start in 300.0s
>>>>> 2019-02-04T22:46:16Z ERROR The ipa-server-upgrade command failed.
See
>>>>> /var/log/ipaupgrade.log for more information
>>>> Thanks,
>>>> -Chris
>>>>
>>>>> Rob,
>>>>>
>>>>> I'll be honest. I think you are suggesting an ldapsearch with
this
>>>>>
>>>>> Check to see which masteris the renewal master. Look in
>>>>> cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
>>>>> ipaConfigString=caRenewalMaster
>>>>>
>>>>> sorry I've not figured out how to successfully ldapsearch
>>>>>
>>>>> Instead I did this:
>>>>> ipa config-show |grep 'CA renewal master'
>>>>>
>>>>> It came up blank. I suspect I didn't have a renewal master
somehow.
>>>>>
>>>>> Then I did This:
>>>>> ipa-csreplica-manage set-renewal-master ipa2 (hostname of working
IPA
>>>>> server)
>>>>>
>>>>> Next is a "yum update" to be safe, and lastly
"ipa-server-upgrade" on
>>>>> ipa2. When that's all done I'll try "yum update"
and
>>>>> "ipa-server-upgrade" on my broken IPA system ipa1
>>>>>
>>>>> I'll report back here when finished.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -Chris
>>>>>
>>>>>> Check to see which masteris the renewal master. Look in
>>>>>> cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
>>>>>> ipaConfigString=caRenewalMaster
>>>>>>
>>>>>> You want to run the script on that master first to get the
certs
>>>>>> renewed.
>>>>>>
>>>>>> I'd start by re-running ipa-server-upgrade. It is idempotent
so
>>>>>> there
>>>>>> should be no risk. It may repair the tracking for you.
>>>>>>
>>>>>> rob
>>>>> On 2/4/2019 3:30 PM, Rob Crittenden wrote:
>>>>>> Chris Mohler via FreeIPA-users wrote:
>>>>>>> Thanks for looking at my issue!
>>>>>>>
>>>>>>> There have been no recent updates on my system. Actually I
was
>>>>>>> getting
>>>>>>> ready to update when I noticed things weren't good.
>>>>>>>
>>>>>>> Here is the output from the log of the most recent update.
Looks
>>>>>>> like it
>>>>>>> was completed successfully. The lines you asked about are
in
>>>>>>> Bold/underlined.
>>>>>>>
>>>>>>>> 2018-07-18T16:55:21Z INFO [Update certmonger certificate
renewal
>>>>>>>> configuration]
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Loading Index file from
>>>>>>>> '/var/lib/ipa/sysrestore/sysrestore.index'
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/usr/bin/certutil -d
>>>>>>>> /etc/pki/pki-tomcat/alias -L -f
>>>>>>>> /etc/pki/pki-tomcat/alias/pwdfile.txt
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return
code=0
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=
>>>>>>>> Certificate Nickname Trust
>>>>>>>> Attributes
>>>>>>>> SSL,S/MIME,JAR/XPI
>>>>>>>>
>>>>>>>> caSigningCert cert-pki-ca CTu,Cu,Cu
>>>>>>>> subsystemCert cert-pki-ca u,u,u
>>>>>>>> ocspSigningCert cert-pki-ca u,u,u
>>>>>>>> auditSigningCert cert-pki-ca u,u,Pu
>>>>>>>> Server-Cert cert-pki-ca u,u,u
>>>>>>>>
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stderr=
>>>>>>>> _*2018-07-18T16:55:21Z DEBUG Configuring certmonger to
stop
>>>>>>>> tracking
>>>>>>>> system certificates for CA*_
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start
>>>>>>>> messagebus.service
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return
code=0
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stderr=
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl
is-active
>>>>>>>> messagebus.service
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return
code=0
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=active
>>>>>>>>
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stderr=
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl start
>>>>>>>> certmonger.service
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return
code=0
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stderr=
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Starting external process
>>>>>>>> 2018-07-18T16:55:21Z DEBUG args=/bin/systemctl
is-active
>>>>>>>> certmonger.service
>>>>>>>> 2018-07-18T16:55:21Z DEBUG Process finished, return
code=0
>>>>>>>> 2018-07-18T16:55:21Z DEBUG stdout=active
>>>>>>>>
>>>>>>> -snip- a few more lines like the section above.
>>>>>>>> 2018-07-18T16:55:25Z DEBUG stderr=
>>>>>>>> 2018-07-18T16:55:30Z DEBUG Loading Index file from
>>>>>>>> '/var/lib/ipa/sysrestore/sysrestore.index'
>>>>>>>> 2018-07-18T16:55:30Z DEBUG Starting external process
>>>>>>>> 2018-07-18T16:55:30Z DEBUG args=/usr/bin/certutil -d
>>>>>>>> /etc/dirsrv/DOMAINNAMEHERE -L -n Server-Cert -a -f
>>>>>>>> /etc/dirsrv/DOMAINNAMEHERE/pwdfile.txt
>>>>>>>> 2018-07-18T16:55:30Z DEBUG Process finished, return
code=0
>>>>>>>> 2018-07-18T16:55:30Z DEBUG stdout=-----BEGIN
CERTIFICATE-----
>>>>>>> -Snip- Cert and Key stuff goes here-
>>>>>>>> 2018-07-18T16:55:34Z DEBUG stderr=
>>>>>>>> _*2018-07-18T16:55:35Z INFO Certmonger certificate
renewal
>>>>>>>> configuration updated*_
>>>>>> Check to see which masteris the renewal master. Look in
>>>>>> cn=CA,cn=$(hostname),cn=masters,cn=ipa,cn=etc,$SUFFIX for
>>>>>> ipaConfigString=caRenewalMaster
>>>>>>
>>>>>> You want to run the script on that master first to get the
certs
>>>>>> renewed.
>>>>>>
>>>>>> I'd start by re-running ipa-server-upgrade. It is idempotent
so
>>>>>> there
>>>>>> should be no risk. It may repair the tracking for you.
>>>>>>
>>>>>> rob
>>>>>>
>>>>>>> On 2/4/2019 1:44 PM, Florence Blanc-Renaud wrote:
>>>>>>>> On 2/4/19 5:59 PM, Chris Mohler via FreeIPA-users
wrote:
>>>>>>>>> Hi Everyone,
>>>>>>>>>
>>>>>>>>> I'm looking for some help. I'm having trouble
with everything
>>>>>>>>> basically.
>>>>>>>>>
>>>>>>>>> I think one of my CA's certs expired or
something. I can't kinit
>>>>>>>>> admin, I can't login via the WebGui. If I
"getcert list" it
>>>>>>>>> returns
>>>>>>>>> "Number of certificates and requests being
tracked: 0."
>>>>>>>>>
>>>>>>>>> This all started happening a few days ago and I am at
a loss
>>>>>>>>> as to
>>>>>>>>> what happened. On a whim I set the system date and
time back a
>>>>>>>>> few
>>>>>>>>> months to see if my certs were expired and like magic
I can
>>>>>>>>> login to
>>>>>>>>> the Webgui but I'm still not tracking anything
with "getcert
>>>>>>>>> list" I
>>>>>>>>> suspect the cert has expired but without tracking it
I can't
>>>>>>>>> tell, or
>>>>>>>>> renew it.
>>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> can you check if an upgrade happened recently (have a
look at
>>>>>>>> /var/log/ipaupgrade.log)? The upgrade stop tracking certs
and
>>>>>>>> re-configures certmonger, so if it failed in the middle
you may be
>>>>>>>> left without any tracking.
>>>>>>>> You should be able to find lines like the following if
the
>>>>>>>> untracking/tracking went fine:
>>>>>>>> ---
>>>>>>>> [Update certmonger certificate renewal configuration]
>>>>>>>> Configuring certmonger to stop tracking system
certificates for CA
>>>>>>>> Certmonger certificate renewal configuration updated
>>>>>>>> ---
>>>>>>>>
>>>>>>>> HTH,
>>>>>>>> flo
>>>>>>>>
>>>>>>>>> Please help
>>>>>>>>>
>>>>>>>>> I'm running Centos 7, FreeIPA 4.5.4
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> -Chris
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> FreeIPA-users mailing list --
>>>>>>>>> freeipa-users(a)lists.fedorahosted.org
>>>>>>>>> To unsubscribe send an email to
>>>>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>>>> Fedora Code of Conduct:
>>>>>>>>>
https://getfedora.org/code-of-conduct.html
>>>>>>>>> List Guidelines:
>>>>>>>>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>>>>> List Archives:
>>>>>>>>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> FreeIPA-users mailing list --
freeipa-users(a)lists.fedorahosted.org
>>>>>>> To unsubscribe send an email to
>>>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>>>> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
>>>>>>> List Guidelines:
>>>>>>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>>>> List Archives:
>>>>>>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>>>>
>>>>>>>
>>>>>>>
>
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...