You'd be right, I worked it out over the weekend. On the fifth time of checking, having convinced myself the certificates all looked good, I renewed the expried Kerberos certificate...

It didn't seem to take effect straight away for bringing up the replica though but I didn't have time to dig in until the next day, and that time it worked first time, so I suspect it was cached somewhere too.

Thanks for the help though! Much appreciated.

Thomas

On Mon, Jun 18, 2018, 11:41 PM Florence Blanc-Renaud <flo@redhat.com> wrote:
Hi Thomas,

you can have a look at
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tomcatd-fails-to-start/

Usually the communication issue between PKI and LDAP is linked to an
expired certificate, or a mismatch between the content of
uid=pkidbuser,ou=people,o=ipaca and the actual certificate.

HTH,
Flo

On 06/15/2018 06:52 AM, Thomas Letherby via FreeIPA-users wrote:
> Hello all,
>
> I'm running FreeIPA on two CentOS 7 servers, one, the master is on a
> physical server, the other (a replica with CA, DNS etc) is running on an
> Ovirt cluster.
>
> I patched the boxes and upgraded IPA on the two servers a few days ago,
> and the master ran through the upgrade without any issue, however the
> replica fails when starting the CA, timing out after the 300 seconds.
> Increasing the timeout to 600 didn't help, and I rebuilt the replica
> from scratch which still gives the same error.
> If I try and restart the services after promoting it it tells me to run
> the upgrade, and if I do so I get the same error as the install:
>
> 2018-06-15T04:48:07Z DEBUG The CA status is: check interrupted due to
> error: Retrieving CA status failed with status 500
> 2018-06-15T04:48:07Z DEBUG Waiting for CA to start...
> 2018-06-15T04:48:08Z DEBUG request POST <replica>:8080
> 2018-06-15T04:48:08Z DEBUG request body ''
> 2018-06-15T04:48:08Z DEBUG response status 500
> 2018-06-15T04:48:08Z DEBUG response headers Server: Apache-Coyote/1.1
> Content-Type: text/html;charset=utf-8
>
> 2018-06-15T04:48:09Z DEBUG The ipa-server-upgrade command failed,
> exception: ScriptError: CA did not start in 300.0s
> 2018-06-15T04:48:09Z ERROR CA did not start in 300.0s
>
> Googling gets me similar problems people have had due to certificate
> expiry, but the dates look good as far as I can see and after a complete
> rebuild it should have issued new ones anyway I think.
>
> Digging through the logs I see variations on the below error, but I'm
> not sure why this would be the case:
> Could not connect to LDAP server host <Replica> port 636 Error
> netscape.ldap.LDAPException: Authentication failed (49)
>
> Browsing to http://<Replica>:8080/ca/admin/ca/getStatus
>
> Gets me this:
>
> *type* Exception report
>
> *message* _Subsystem unavailable_
>
> *description* _The server encountered an internal error that prevented
> it from fulfilling this request._
>
> *exception*
>
> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
>       com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
>       org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
>       org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>       org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
>       org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
>       org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
>       org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
>       org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
>       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>       java.lang.Thread.run(Thread.java:748)
>
> *note* _The full stack trace of the root cause is available in the
> Apache Tomcat/7.0.76 logs._
>
>
> I'm a bit stuck as to how to proceed fixing this, I'm not overly
> familiar with what logs do what with IPA, and I'm not seeing anything
> obviously wrong with the configuration.
>
> Has anyone seen this before, or can point me in the right direction to
> track this down?
>
> Thanks,
>
> Thomas
>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorahosted.org/message/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
>