Hi Rob,
My instinct was to look in the /var/log/pki/pki-tomcat/ca/*.debug but those files are
dated since tomcat has been broken for a couple of weeks. Essentially it doesn't
generate new logs since it fails to start and I can't use them for debugging.
This is the output of
journalctl -u pki-tomcatd(a)pki-tomcat.service
august 16 16:03:45 ipa.domain.priv systemd[1]: Starting PKI Tomcat Server pki-tomcat...
august 16 16:04:05 ipa.domain.priv server[168252]: Java virtual machine used:
/usr/lib/jvm/jre-1.8.0-openjdk/bin/java
august 16 16:04:05 ipa.domain.priv server[168252]: classpath used:
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar
august 16 16:04:05 ipa.domain.priv server[168252]: main class used:
org.apache.catalina.startup.Bootstrap
august 16 16:04:05 ipa.domain.priv server[168252]: flags used: -Dcom.redhat.fips=false
august 16 16:04:05 ipa.domain.priv server[168252]: options used:
-Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat
-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.security.manager
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
august 16 16:04:05 ipa.domain.priv server[168252]: arguments used: start
august 16 16:04:10 ipa.domain.priv ipa-pki-wait-running[168253]: pki.client:
/usr/libexec/ipa/ipa-pki-wait-running:64: The subsystem in PKIConnection.__init__() has
been deprecated (
https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes).
august 16 16:04:11 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Created connection
http://ipa.domain.priv:8080/ca
august 16 16:04:11 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:12 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:13 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:14 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:15 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:16 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:17 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:18 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:19 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:20 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:21 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:22 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:23 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:24 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:25 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:26 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:27 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:28 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:29 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:30 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:31 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:32 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:33 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:34 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:35 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:36 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:37 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:39 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:40 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:41 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:42 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:43 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:44 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:45 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:46 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:47 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:48 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:49 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:50 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:51 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:52 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:53 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:54 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:55 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:56 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:57 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:58 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:04:59 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:00 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:01 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:02 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:03 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:04 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:05 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:06 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:07 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:09 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:10 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:11 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:12 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:13 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:14 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:15 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:16 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:17 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:18 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:19 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:20 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:21 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:22 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:23 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:24 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:25 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:26 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:27 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:28 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:29 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:30 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:31 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:32 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:33 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:34 ipa.domain.priv ipa-pki-wait-running[168253]: ipa-pki-wait-running:
Request failed unexpectedly, 404 Client Error: for url:
http://ipa.domain.priv:8080/ca/admin/ca/getStatus
august 16 16:05:35 ipa.domain.priv systemd[1]: pki-tomcatd(a)pki-tomcat.service: Start-post
operation timed out. Stopping.
august 16 16:07:05 ipa.domain.priv systemd[1]: pki-tomcatd(a)pki-tomcat.service: State
'stop-sigterm' timed out. Killing.
august 16 16:07:05 ipa.domain.priv systemd[1]: pki-tomcatd(a)pki-tomcat.service: Killing
process 168252 (java) with signal SIGKILL.
august 16 16:07:05 ipa.domain.priv systemd[1]: pki-tomcatd(a)pki-tomcat.service: Main
process exited, code=killed, status=9/KILL
august 16 16:07:05 ipa.domain.priv systemd[1]: pki-tomcatd(a)pki-tomcat.service: Failed with
result 'timeout'.
august 16 16:07:05 ipa.domain.priv systemd[1]: Failed to start PKI Tomcat Server
pki-tomcat.
IPA runs in a kvm/qemu VM, the iptables look fine:
iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
LIBVIRT_PRT all -- anywhere anywhere
Chain LIBVIRT_PRT (1 references)
target prot opt source destination
RETURN all -- 192.168.122.0/24
base-address.mcast.net/24
RETURN all -- 192.168.122.0/24 255.255.255.255
MASQUERADE tcp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE udp -- 192.168.122.0/24 !192.168.122.0/24 masq ports: 1024-65535
MASQUERADE all -- 192.168.122.0/24 !192.168.122.0/24
Where else would I be looking?
Cheers.