Hello all,

I'm running FreeIPA on two CentOS 7 servers, one, the master is on a physical server, the other (a replica with CA, DNS etc) is running on an Ovirt cluster.

I patched the boxes and upgraded IPA on the two servers a few days ago, and the master ran through the upgrade without any issue, however the replica fails when starting the CA, timing out after the 300 seconds. Increasing the timeout to 600 didn't help, and I rebuilt the replica from scratch which still gives the same error.
If I try and restart the services after promoting it it tells me to run the upgrade, and if I do so I get the same error as the install:

2018-06-15T04:48:07Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500
2018-06-15T04:48:07Z DEBUG Waiting for CA to start...
2018-06-15T04:48:08Z DEBUG request POST <replica>:8080
2018-06-15T04:48:08Z DEBUG request body ''
2018-06-15T04:48:08Z DEBUG response status 500
2018-06-15T04:48:08Z DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8

2018-06-15T04:48:09Z DEBUG The ipa-server-upgrade command failed, exception: ScriptError: CA did not start in 300.0s
2018-06-15T04:48:09Z ERROR CA did not start in 300.0s

Googling gets me similar problems people have had due to certificate expiry, but the dates look good as far as I can see and after a complete rebuild it should have issued new ones anyway I think.

Digging through the logs I see variations on the below error, but I'm not sure why this would be the case:
Could not connect to LDAP server host <Replica> port 636 Error netscape.ldap.LDAPException: Authentication failed (49)

Browsing to http://<Replica>:8080/ca/admin/ca/getStatus

Gets me this:

type Exception report

message Subsystem unavailable

description The server encountered an internal error that prevented it from fulfilling this request.

exception

javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
	com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
	org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
	org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
	org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
	org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
	org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
	org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
	java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.lang.Thread.run(Thread.java:748)

note The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.


I'm a bit stuck as to how to proceed fixing this, I'm not overly familiar with what logs do what with IPA, and I'm not seeing anything obviously wrong with the configuration. 

Has anyone seen this before, or can point me in the right direction to track this down?

Thanks,

Thomas