Hi,
it looks like some of the certificates used by PKI are also expired (they
are stored in /etc/pki/pki-tomcat/alias). Since you're running IPA 4.9, you
can use the command ipa-cert-fix. Please read the man page with extra care,
it recommends to backup certificates and keys before you proceed.
You mentioned having a pair of IPA servers, do they both have expired
certificates? If one of them is good, there are also other options to
retrieve the renewed certificates from the good server and install them on
the other one (the 3 certs ocspSigningCert, subsystemCert and
auditSigningCert are shared on all the CA instances).
flo
On Fri, Mar 11, 2022 at 2:36 PM Morgan Cox via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org> wrote:
Hi
We have a pair of Freeipa (4.9.x) Rhel8 Freeipa servers.
Previously we had installed a 3rd party cert for httpd + dirsrv (only) -
this expired recently. I was unable to login to ui . This issue however may
not be connected with this. It appears to be linked to Tomcat -> LDAPS
connectiopn ?? - error when trying to login was 'Login failed due to an
unknown reason'
I could login if I changed server time to the past - but the certificates
page is broken 'Certificate operation cannot be completed: Unable to
communicate with CMS (503)' (time has been set back to normal now)
As a result I cannot renew my httpd/dirsv cert
Can anyone help me restore pki-tomcatd ? This may not be connected to
web/dirsv cert expiry (and just be a coincidence)
If I try using
# ipa-server-certinstall --http --dirsrv ireland.idm.domain.uk.key
ireland.idm.domain.uk.crt
I get
-----
Directory Manager password:
Enter private key unlock password:
cannot connect to 'https://london.idm.domain.uk:443/acme/directory':
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
The ipa-server-certinstall command failed.
----
I can however install the cert to just the dirsv
---
[root@london mcox]# ipa-server-certinstall --dirsrv
london.idm.domain.uk.key london.idm.domain.uk.crt
Directory Manager password:
Enter private key unlock password:
Please restart ipa services after installing certificate (ipactl restart)
---
However after ipactl restart -> pki-tomcatd Service: STOPPED (all other
services are working)
The main IPA system aside from this appears to work - i.e I can login and
sudo to clients, and kinit, etc works
As a work-around I can login to the UI if I manually copy the cert/key to
/var/lib/ipa/certs/httpd.crt
/var/lib/ipa/private/httpd.key
However the pki-tomcatd service is still down - I see these errors
- On certifcates tab : IPA Error 4301: CertificateOperationError -
Certificate operation cannot be completed: Unable to communicate with CMS
(503)
- On Certificate authorities pages I see : Some operations failed ->
details -> Failed to authenticate to CA REST API
pki-tomcatd logs show
-------
Mar 11 12:54:10 london.idm.domain.uk systemd[1]: Starting PKI Tomcat
Server pki-tomcat...
Mar 11 12:54:15 london.idm.domain.uk server[509585]: Java virtual machine
used: /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
Mar 11 12:54:15 london.idm.domain.uk server[509585]: classpath used:
/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/ant.jar:/usr/share/java/ant-launcher.jar:/usr/lib/jvm/java/lib/tools.jar
Mar 11 12:54:15 london.idm.domain.uk server[509585]: main class used:
org.apache.catalina.startup.Bootstrap
Mar 11 12:54:15 london.idm.domain.uk server[509585]: flags used:
-Dcom.redhat.fips=false
Mar 11 12:54:15 london.idm.domain.uk server[509585]: options used:
-Dcatalina.base=/var/lib/pki/pki-tomcat -Dcatalina.home=/usr/share/tomcat
-Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
-Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djava.security.manager
-Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
Mar 11 12:54:15 london.idm.domain.uk server[509585]: arguments used: start
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]:
pki.client: /usr/libexec/ipa/ipa-pki-wait-running:63: The subsystem in
PKIConnection.__init__() has been deprecated (
https://www.dogtagpki.org/wiki/PKI_10.8_Python_Changes).
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Created connection
http://london.idm.domain.uk:8080/ca
Mar 11 12:54:16 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='
london.idm.domain.uk', port=8080): Max retries exceeded with url:
/ca/admin/ca/getStatus (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7fb6c264e668>: Failed to establish a new connection: [Errno 111]
Connection refused',))
Mar 11 12:54:17 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='
london.idm.domain.uk', port=8080): Max retries exceeded with url:
/ca/admin/ca/getStatus (Caused by
NewConnectionError('<urllib3.connection.HTTPConnection object at
0x7fb6c264ea58>: Failed to establish a new connection: [Errno 111]
Connection refused',))
Mar 11 12:54:18 london.idm.domain.uk server[509585]: WARNING: Some of the
specified [protocols] are not supported by the SSL engine and have been
skipped: [[TLSv1, TLSv1.1]]
Mar 11 12:54:19 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='
london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:21 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='
london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: One or more
listeners failed to start. Full details will be found in the appropriate
container log file
Mar 11 12:54:23 london.idm.domain.uk server[509585]: SEVERE: Context
[/ca] startup failed due to previous errors
Mar 11 12:54:23 london.idm.domain.uk server[509585]: WARNING: The web
application [ca] appears to have started a thread named [LDAPConnThread-0
ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very
likely to create a memory leak. Stack trace of thread:
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.socketRead0(Native Method)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.read(SocketInputStream.java:171)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.read(SocketInputStream.java:141)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.read(SocketInputStream.java:127)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
netscape.ldap.ber.stream.BERElement.getElement(Unknown Source)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
netscape.ldap.LDAPConnThread.run(Unknown Source)
Mar 11 12:54:23 london.idm.domain.uk server[509585]:
java.lang.Thread.run(Thread.java:748)
Mar 11 12:54:23 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Connection failed: HTTPConnectionPool(host='
london.idm.domain.uk', port=8080): Read timed out. (read timeout=1.0)
Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: One or more
listeners failed to start. Full details will be found in the appropriate
container log file
Mar 11 12:54:24 london.idm.domain.uk server[509585]: SEVERE: Context
[/acme] startup failed due to previous errors
Mar 11 12:54:24 london.idm.domain.uk server[509585]: WARNING: The web
application [acme] appears to have started a thread named [LDAPConnThread-1
ldaps://london.idm.domain.uk:636] but has failed to stop it. This is very
likely to create a memory leak. Stack trace of thread:
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.socketRead0(Native Method)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.read(SocketInputStream.java:171)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.read(SocketInputStream.java:141)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
java.net.SocketInputStream.read(SocketInputStream.java:127)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
org.mozilla.jss.ssl.SSLSocket.socketRead(Native Method)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
org.mozilla.jss.ssl.SSLSocket.read(SSLSocket.java:1505)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
org.mozilla.jss.ssl.SSLInputStream.read(SSLInputStream.java:43)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
java.io.BufferedInputStream.read(BufferedInputStream.java:265)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
netscape.ldap.ber.stream.BERElement.getElement(Unknown Source)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
netscape.ldap.LDAPConnThread.run(Unknown Source)
Mar 11 12:54:24 london.idm.domain.uk server[509585]:
java.lang.Thread.run(Thread.java:748)
Mar 11 12:54:25 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for
url:
http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
Mar 11 12:54:26 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for
url:
http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
Mar 11 12:54:27 london.idm.domain.uk ipa-pki-wait-running[509586]:
ipa-pki-wait-running: Request failed unexpectedly, 404 Client Error: for
url:
http://london.idm.domain.uk:8080/ca/admin/ca/getStatus
...
-------
Other logs show : (i've just added the main error - not entire java error
/var/log/pki/pki-tomcat/acme/debug.2022-03-11.log :
-----
12:34:01 [main] SEVERE: Exception sending context initialized event to
listener instance of class [org.dogtagpki.acme.server.ACMEEngine]
java.lang.RuntimeException: Unable to start ACME engine: Unable to connect
to LDAP server: Authentication failed
-----
/var/log/pki/pki-tomcat/ca/debug.2022-03-11.log :
-----
2022-03-11 12:33:59 [main] SEVERE: Unable to start CA engine: Unable to
connect to LDAP server: Authentication failed
Unable to connect to L2022-03-11 12:33:59 [main] INFO: Shutting down CA
subsystem
....
2022-03-11 12:33:59 [main] SEVERE: Exception sending context destroyed
event to listener instance of class [org.dogtagpki.server.ca.CAEngine]
java.lang.NullPointerException
DAP server: Authentication failed
-----
I have checked this ->
# getcert list |grep expire
expires: 2024-02-13 00:32:37 GMT
expires: unknown
expires: unknown
expires: unknown
expires: unknown
expires: 2024-01-22 00:29:51 GMT
expires: 2024-01-22 00:30:38 GMT
And I have ran ipa-healthcheck
I can see
----
Expired Cert: ocsp_signing
Expired Cert: subsystem
Expired Cert: audit_signing
Internal server error 503 Server Error: Service Unavailable for url:
http://london.idm.domain.uk:80/ca/rest/securityDomain/domainInfo
Internal server error HTTPSConnectionPool(host='london.idm.domain.uk',
port=8443): Max retries exceeded with url: /ca/admin/ca/getStatus (Caused
by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object
at 0x7fc4e6c58198>: Failed to establish a new connection: [Errno 111]
Connection refused',))
---
Also some expired certs
"source": "pki.server.healthcheck.certs.expiration",
"check": "CASystemCertExpiryCheck",
"result": "ERROR",
"uuid": "36c8fbed-571d-4d38-9919-53322fea4aa2",
"when": "20220311130832Z",
"duration": "0.188329",
"kw": {
"cert_id": "ocsp_signing",
"expiry_date": "Mar 01 2022",
"msg": "Certificate has ALREADY EXPIRED"
}
},
{
"source": "pki.server.healthcheck.certs.expiration",
"check": "CASystemCertExpiryCheck",
"result": "ERROR",
"uuid": "195970e4-e2fd-4eca-aeac-f1e97e9c3b13",
"when": "20220311130832Z",
"duration": "0.360146",
"kw": {
"cert_id": "subsystem",
"expiry_date": "Mar 01 2022",
"msg": "Certificate has ALREADY EXPIRED"
}
},
{
"source": "pki.server.healthcheck.certs.expiration",
"check": "CASystemCertExpiryCheck",
"result": "ERROR",
"uuid": "a84a9bc5-de4d-4cdc-b7fd-41b83f3a11af",
"when": "20220311130833Z",
"duration": "0.454225",
"kw": {
"cert_id": "audit_signing",
"expiry_date": "Mar 01 2022",
"msg": "Certificate has ALREADY EXPIRED"
}
I have attached the full output of healthcheck to :
https://pastebin.com/xfNLR0Ja (domain name changed)
On the last ipa update there was also a issue with pki-tomcatd - i.e - I
have to remove the block 'requiredSecret=' in
/etc/pki/pki-tomcat/server.xml to fix it, this was however working for a
month or so after .
Any help to troubleshooting this would be welcomed
Thanks
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure