From manuel.gugliucci at scaiitec.it Mon Feb 8 10:59:46 2021 Content-Type: multipart/mixed; boundary="===============5892905460819013252==" MIME-Version: 1.0 From: Manuel Gugliucci To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] IPA certs expired, pki-tomcatd fails to start Date: Mon, 08 Feb 2021 10:59:36 +0000 Message-ID: <20210208105936.31782.85508@mailman01.iad2.fedoraproject.org> --===============5892905460819013252== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hello, I'm running a freeipa server over a cloudera cluster, on 2020-12-31 all the= certs expired and did not renew by itself. After I set the system date before the expiration date, I tried ipa-cacert-= renew but returns an error saying that ca cert are not managed by certmonge= r so I did a getcert resubmit for every cert. Almos all went on "Monitoring" state, except for one that says "NEED_CSR_GE= N_PIN". If I try to do 'ipactl start', it starts to first upgrade IPA and fails bec= ause of the pki-tomcat service: ``` 2019-12-31T19:12:01Z DEBUG response body 'Apache Tomcat/= 7.0.76 - Error report

HTTP Status 500 - Sub= system unavailable


type Ex= ception report

message Subsystem unavailable

d= escription The server encountered an internal error that prevented i= t from fulfilling this requ est.

exception

javax.ws.rs.ServiceUnavailableExcepti=
on: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurit=
yConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Auth=
enticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.val=
ves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catali=
na.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.cata=
lina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.=
coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:=
1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.proce=
ss(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$Soc=
ketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolEx=
ecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.Thre=
adPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat=
.util.threads.TaskThrea
 d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.=
java:748)\n

note The full stack trace of the root cau= se is available in the Apache Tomcat/7.0.76 logs.


Apache Tomcat/7.0.76

' 2019-12-31T19:12:01Z DEBUG The CA status is: check interrupted due to error= : Retrieving CA status failed with status 500 2019-12-31T19:12:01Z DEBUG Waiting for CA to start... ``` I also looked for the previous threads listed on this forum, but none of th= em provided a solution --===============5892905460819013252==-- From flo at redhat.com Mon Feb 8 12:53:29 2021 Content-Type: multipart/mixed; boundary="===============2472195656912242657==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Mon, 08 Feb 2021 13:53:07 +0100 Message-ID: <1a01c8c7-1a97-0fc0-db36-07c08a37bb63@redhat.com> In-Reply-To: 20210208105936.31782.85508@mailman01.iad2.fedoraproject.org --===============2472195656912242657== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 2/8/21 11:59 AM, Manuel Gugliucci via FreeIPA-users wrote: > Hello, > = > I'm running a freeipa server over a cloudera cluster, on 2020-12-31 all t= he certs expired and did not renew by itself. > = > After I set the system date before the expiration date, I tried ipa-cacer= t-renew but returns an error saying that ca cert are not managed by certmon= ger so I did a getcert resubmit for every cert. Hi, the command "ipa-cacert-manage renew" is used to renew the CA = certificate (in case IPA was installed with an embedded CA), not the = other certs. Before giving any advice, I would like to know more about the = deployment. Are there a single or multiple IPA servers? Is the CA role = deployed on multiple servers? Which version is installed? # kinit admin; ipa server-role-find # rpm -qa *ipa-server Which certificates are valid or expired? # getcert list Thanks, flo > = > Almos all went on "Monitoring" state, except for one that says "NEED_CSR_= GEN_PIN". > = > If I try to do 'ipactl start', it starts to first upgrade IPA and fails b= ecause of the pki-tomcat service: > = > ``` > 2019-12-31T19:12:01Z DEBUG response body 'Apache Tomca= t/7.0.76 - Error report

HTTP Status 500 - S= ubsystem unavailable


type = Exception report

message Subsystem unavailable

description The server encountered an internal error that prevented= it from fulfilling this requ > est.

exception

javax.ws.rs.ServiceUnavailableExce=
ption: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecu=
rityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.A=
uthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.=
valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.cat=
alina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.c=
atalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apac=
he.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.ja=
va:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.pr=
ocess(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$=
SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoo=
lExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.T=
hreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tom=
cat.util.threads.TaskThrea
>   d$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre=
ad.java:748)\n

note The full stack trace of the root = cause is available in the Apache Tomcat/7.0.76 logs.


Apache Tomcat/7.0.76

' > 2019-12-31T19:12:01Z DEBUG The CA status is: check interrupted due to err= or: Retrieving CA status failed with status 500 > 2019-12-31T19:12:01Z DEBUG Waiting for CA to start... > ``` > I also looked for the previous threads listed on this forum, but none of = them provided a solution > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code= -of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users= (a)lists.fedorahosted.org > = --===============2472195656912242657==-- From manuel.gugliucci at scaiitec.it Mon Feb 8 13:04:01 2021 Content-Type: multipart/mixed; boundary="===============0930619118656160908==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Mon, 08 Feb 2021 13:03:51 +0000 Message-ID: <20210208130351.21528.80269@mailman01.iad2.fedoraproject.org> In-Reply-To: 1a01c8c7-1a97-0fc0-db36-07c08a37bb63@redhat.com --===============0930619118656160908== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Florence, thanks for the answer it's a single IPA server, VERSION: 4.6.8, API_VERSION: 2.237 I kinit as admin without problems, then: [root(a)ipa1 ~]# ipa server-role-find ipa: ERROR: cannot connect to 'https://ipa1.itec.lab/ipa/json': Internal Se= rver Error [root(a)ipa1 ~]# rpm -qa *ipa-server ipa-server-4.6.8-5.el7.centos.x86_64 # getcert list Number of certificates and requests being tracked: 7. Request ID '20191231201955': status: MONITORING stuck: no key pair storage: type=3DFILE,location=3D'/var/kerberos/krb5kdc/kdc.key' certificate: type=3DFILE,location=3D'/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=3Dipa1.itec.lab,O=3DITEC.LAB subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB expires: 2020-12-31 20:19:55 UTC principal name: krbtgt/ITEC.LAB(a)ITEC.LAB certificate template/profile: KDCs_PKINIT_Certs pre-save command: = post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20201102185036': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DCA Audit,O=3DITEC.LAB expires: 2020-12-08 09:35:14 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigning= Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185037': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DOCSP Subsystem,O=3DITEC.LAB expires: 2020-12-08 09:38:07 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningC= ert cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185038': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DCA Subsystem,O=3DITEC.LAB expires: 2020-12-08 09:37:36 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCer= t cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185039': status: NEED_CSR_GEN_PIN stuck: yes key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent-selfsigned issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DCertificate Authority,O=3DITEC.LAB expires: 2037-01-25 14:22:25 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCer= t cert-pki-ca" track: yes auto-renew: yes Request ID '20201102185040': status: MONITORING stuck: no key pair storage: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.key' certificate: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DIPA RA,O=3DITEC.LAB expires: 2020-12-08 09:37:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20201102185042': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB expires: 2020-12-08 09:35:05 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert = cert-pki-ca" track: yes auto-renew: yes I had to set my date in several weeks before the expiring to renew them via= certmonger, but it does not auto-renew past 30-12-2020 Thanks for the support, Manuel --===============0930619118656160908==-- From flo at redhat.com Mon Feb 8 13:47:39 2021 Content-Type: multipart/mixed; boundary="===============7017014442913194653==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Mon, 08 Feb 2021 14:47:20 +0100 Message-ID: In-Reply-To: 20210208130351.21528.80269@mailman01.iad2.fedoraproject.org --===============7017014442913194653== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 2/8/21 2:03 PM, Manuel Gujo via FreeIPA-users wrote: > Hi Florence, thanks for the answer > = > it's a single IPA server, VERSION: 4.6.8, API_VERSION: 2.237 > = Hi, The CA is self-signed and still valid, and you are lucky because this = ipa version already provides a new tool called ipa-cert-fix that should = be able to help renew the certificates. For more information please refer to the doc [1]. ipa-cert-fix analyzes = the existing certificates and lists the ones that need to be renewed, = then prompts you for confirmation and proceeds. No need to move the date = in the past or do manual steps. HTH, flo [1] = https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/ht= ml/linux_domain_identity_authentication_and_policy_guide/cert-renewal#renew= ing-expired-system-certificate-when-idm-is-offline > I kinit as admin without problems, then: > = > [root(a)ipa1 ~]# ipa server-role-find > ipa: ERROR: cannot connect to 'https://ipa1.itec.lab/ipa/json': Internal = Server Error > [root(a)ipa1 ~]# rpm -qa *ipa-server > ipa-server-4.6.8-5.el7.centos.x86_64 > = > # getcert list > Number of certificates and requests being tracked: 7. > Request ID '20191231201955': > status: MONITORING > stuck: no > key pair storage: type=3DFILE,location=3D'/var/kerberos/krb5kdc/kdc.key' > certificate: type=3DFILE,location=3D'/var/kerberos/krb5kdc/kdc.crt' > CA: SelfSign > issuer: CN=3Dipa1.itec.lab,O=3DITEC.LAB > subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB > expires: 2020-12-31 20:19:55 UTC > principal name: krbtgt/ITEC.LAB(a)ITEC.LAB > certificate template/profile: KDCs_PKINIT_Certs > pre-save command: > post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert > track: yes > auto-renew: yes > Request ID '20201102185036': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nicknam= e=3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3DITEC.LAB > subject: CN=3DCA Audit,O=3DITEC.LAB > expires: 2020-12-08 09:35:14 UTC > key usage: digitalSignature,nonRepudiation > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigni= ngCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20201102185037': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nicknam= e=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3DITEC.LAB > subject: CN=3DOCSP Subsystem,O=3DITEC.LAB > expires: 2020-12-08 09:38:07 UTC > eku: id-kp-OCSPSigning > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSignin= gCert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20201102185038': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB',pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nicknam= e=3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3DITEC.LAB > subject: CN=3DCA Subsystem,O=3DITEC.LAB > expires: 2020-12-08 09:37:36 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherm= ent > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemC= ert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20201102185039': > status: NEED_CSR_GEN_PIN > stuck: yes > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nicknam= e=3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent-selfsigned > issuer: CN=3DCertificate Authority,O=3DITEC.LAB > subject: CN=3DCertificate Authority,O=3DITEC.LAB > expires: 2037-01-25 14:22:25 UTC > key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningC= ert cert-pki-ca" > track: yes > auto-renew: yes > Request ID '20201102185040': > status: MONITORING > stuck: no > key pair storage: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.key' > certificate: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.pem' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3DITEC.LAB > subject: CN=3DIPA RA,O=3DITEC.LAB > expires: 2020-12-08 09:37:47 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherm= ent > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre > post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert > track: yes > auto-renew: yes > Request ID '20201102185042': > status: MONITORING > stuck: no > key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',ni= ckname=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin set > certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nicknam= e=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' > CA: dogtag-ipa-ca-renew-agent > issuer: CN=3DCertificate Authority,O=3DITEC.LAB > subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB > expires: 2020-12-08 09:35:05 UTC > key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherm= ent > eku: id-kp-serverAuth,id-kp-clientAuth > pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad > post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cer= t cert-pki-ca" > track: yes > auto-renew: yes > = > I had to set my date in several weeks before the expiring to renew them v= ia certmonger, but it does not auto-renew past 30-12-2020 > = > Thanks for the support, > Manuel > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code= -of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users= (a)lists.fedorahosted.org >=20 --===============7017014442913194653==-- From manuel.gugliucci at scaiitec.it Mon Feb 8 13:56:42 2021 Content-Type: multipart/mixed; boundary="===============6890672297103561626==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Mon, 08 Feb 2021 13:56:33 +0000 Message-ID: <20210208135633.18730.11573@mailman01.iad2.fedoraproject.org> In-Reply-To: ca050885-b5c8-c67e-acde-c8f451cf7bdc@redhat.com --===============6890672297103561626== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi, I re-sync the date to today and ran ipa-cert-fix but it returns an error [root(a)ipa1 ~]# ipa-cert-fix WARNING ipa-cert-fix is intended for recovery when expired certificates prevent the normal operation of IPA. It should ONLY be used in such scenarios, and backup of the system, especially certificates and keys, is STRONGLY RECOMMENDED. The following certificates will be renewed: = Dogtag sslserver certificate: Subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB Serial: 17 Expires: 2020-12-08 09:35:05 Dogtag subsystem certificate: Subject: CN=3DCA Subsystem,O=3DITEC.LAB Serial: 19 Expires: 2020-12-08 09:37:36 Dogtag ca_ocsp_signing certificate: Subject: CN=3DOCSP Subsystem,O=3DITEC.LAB Serial: 21 Expires: 2020-12-08 09:38:07 Dogtag ca_audit_signing certificate: Subject: CN=3DCA Audit,O=3DITEC.LAB Serial: 18 Expires: 2020-12-08 09:35:14 IPA IPA RA certificate: Subject: CN=3DIPA RA,O=3DITEC.LAB Serial: 20 Expires: 2020-12-08 09:37:47 IPA Apache HTTPS certificate: Subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB Serial: 24 Expires: 2020-12-30 09:35:04 IPA LDAP certificate: Subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB Serial: 25 Expires: 2020-12-30 09:35:16 IPA KDC certificate: Subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB Serial: 1 Expires: 2020-12-31 20:19:55 Enter "yes" to proceed: yes Proceeding. [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver.c= rt' The ipa-cert-fix command failed. --===============6890672297103561626==-- From flo at redhat.com Tue Feb 9 08:36:49 2021 Content-Type: multipart/mixed; boundary="===============6556618621630545301==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Tue, 09 Feb 2021 09:31:19 +0100 Message-ID: In-Reply-To: 20210208135633.18730.11573@mailman01.iad2.fedoraproject.org --===============6556618621630545301== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 2/8/21 2:56 PM, Manuel Gujo via FreeIPA-users wrote: > Hi, > = > I re-sync the date to today and ran ipa-cert-fix but it returns an error > = > [root(a)ipa1 ~]# ipa-cert-fix > = > WARNING > = > ipa-cert-fix is intended for recovery when expired certificates > prevent the normal operation of IPA. It should ONLY be used > in such scenarios, and backup of the system, especially certificates > and keys, is STRONGLY RECOMMENDED. > = > = > The following certificates will be renewed: > = > Dogtag sslserver certificate: > Subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB > Serial: 17 > Expires: 2020-12-08 09:35:05 > = > Dogtag subsystem certificate: > Subject: CN=3DCA Subsystem,O=3DITEC.LAB > Serial: 19 > Expires: 2020-12-08 09:37:36 > = > Dogtag ca_ocsp_signing certificate: > Subject: CN=3DOCSP Subsystem,O=3DITEC.LAB > Serial: 21 > Expires: 2020-12-08 09:38:07 > = > Dogtag ca_audit_signing certificate: > Subject: CN=3DCA Audit,O=3DITEC.LAB > Serial: 18 > Expires: 2020-12-08 09:35:14 > = > IPA IPA RA certificate: > Subject: CN=3DIPA RA,O=3DITEC.LAB > Serial: 20 > Expires: 2020-12-08 09:37:47 > = > IPA Apache HTTPS certificate: > Subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB > Serial: 24 > Expires: 2020-12-30 09:35:04 > = > IPA LDAP certificate: > Subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB > Serial: 25 > Expires: 2020-12-30 09:35:16 > = > IPA KDC certificate: > Subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB > Serial: 1 > Expires: 2020-12-31 20:19:55 > = > Enter "yes" to proceed: yes > Proceeding. > [Errno 2] No such file or directory: '/etc/pki/pki-tomcat/certs/sslserver= .crt' > The ipa-cert-fix command failed. Hi, which version of pki-server is installed? You may be hitting = https://bugzilla.redhat.com/show_bug.cgi?id=3D1897120 Looks like you will need to manually fix the renewal issue by following = the old good method with changing date etc... The first expiration date is 2020-12-08, the system date needs to be = moved before that date. Please try and let me know if there are issues. flo > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code= -of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users= (a)lists.fedorahosted.org >=20 --===============6556618621630545301==-- From manuel.gugliucci at scaiitec.it Tue Feb 9 09:06:28 2021 Content-Type: multipart/mixed; boundary="===============4807286496341520518==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Tue, 09 Feb 2021 09:06:18 +0000 Message-ID: <20210209090618.13571.83744@mailman01.iad2.fedoraproject.org> In-Reply-To: d0c8d7da-7814-e0f0-b615-b64b10102f47@redhat.com --===============4807286496341520518== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi, pki-server version is 10.5.17, if I hit pki-server-upgrade it says this: # pki-server-upgrade Upgrading from version 10.5.17 to 10.5.18: 1. Fix EC admin certificate profile (Yes/No) [Y]: could be helpful? I also found this: https://www.dogtagpki.org/wiki/Tomcat_SSL_Configuration_= with_OpenSSL could generating a self signed sslserver.crt a good idea? in the bugzilla link you sent, do I have to download and run that pyscript? About the old method with turn the date before expiration, what I have to d= o precisely? Because I already done this: #getcert list //list all the cert IDs #ipa-getcert resubmit -i //for every cert listed before #systemctl certmonger restart Thanks for the support Flo Manuel --===============4807286496341520518==-- From manuel.gugliucci at scaiitec.it Fri Feb 12 09:05:20 2021 Content-Type: multipart/mixed; boundary="===============2927378700239064078==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Fri, 12 Feb 2021 09:05:10 +0000 Message-ID: <20210212090510.25276.17909@mailman01.iad2.fedoraproject.org> In-Reply-To: d0c8d7da-7814-e0f0-b615-b64b10102f47@redhat.com --===============2927378700239064078== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi, I've retried to move date three weeks before 2020-12-08 and renew cert manu= ally # ipa-getcert resubmit -i "ID" Resubmitting "20201102185036" to "dogtag-ipa-ca-renew-agent". Here's one of the output log from journalctl -xe # journalctl -xe nov 17 18:08:27 ipa1.itec.lab certmonger[27108]: 2020-11-17 18:08:27 [27108= ] Internal error nov 17 18:08:29 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[28053]: Trac= eback (most recent call last): Fi= le "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in= = sys.exit(main()) Fi= le "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 507, in= main = kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) Fi= le "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, in = kinit_keytab = cred =3D gssapi.Credentials(name=3Dname, store=3Dstore, usage=3D'initiate') Fi= le "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __new__ = store=3Dstore) Fi= le "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acqui= re = usage) Fi= le "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_cre= d_from (gssapi/raw/ext_cred GSSE= rror: Major (851968): Unspecified GSS failure. Minor code may provide more= information, Minor (252963 now all the certs (except from kerberos and CA ones) are status: CA_UNREACH= ABLE. CA cert is status: NEED_CSR_GEN_PIN=20 --===============2927378700239064078==-- From rcritten at redhat.com Fri Feb 12 15:37:21 2021 Content-Type: multipart/mixed; boundary="===============5128773531211025706==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Fri, 12 Feb 2021 10:37:05 -0500 Message-ID: <4d3a716a-4dae-6e55-a450-833da4daf128@redhat.com> In-Reply-To: 20210212090510.25276.17909@mailman01.iad2.fedoraproject.org --===============5128773531211025706== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Manuel Gujo via FreeIPA-users wrote: > Hi, > = > I've retried to move date three weeks before 2020-12-08 and renew cert ma= nually > = > # ipa-getcert resubmit -i "ID" > Resubmitting "20201102185036" to "dogtag-ipa-ca-renew-agent". > = > Here's one of the output log from journalctl -xe > = > # journalctl -xe > nov 17 18:08:27 ipa1.itec.lab certmonger[27108]: 2020-11-17 18:08:27 [271= 08] Internal error > nov 17 18:08:29 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[28053]: Tr= aceback (most recent call last): > = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, = in > = sys.exit(main()) > = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 507, = in main > = kinit_keytab(principal, paths.KRB5_KEYTAB, ccache_filename) > = File "/usr/lib/python2.7/site-packages/ipalib/install/kinit.py", line 47, i= n kinit_keytab > = cred =3D gssapi.Credentials(name=3Dname, store=3Dstore, usage=3D'initiate= ') > = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 64, in __ne= w__ > = store=3Dstore) > = File "/usr/lib64/python2.7/site-packages/gssapi/creds.py", line 148, in acq= uire > = usage) > = File "ext_cred_store.pyx", line 182, in gssapi.raw.ext_cred_store.acquire_c= red_from (gssapi/raw/ext_cred > GS= SError: Major (851968): Unspecified GSS failure. Minor code may provide mo= re information, Minor (252963 > = > now all the certs (except from kerberos and CA ones) are status: CA_UNREA= CHABLE. > = > CA cert is status: NEED_CSR_GEN_PIN = When you are moving back in time are you bringing the IPA services back up? You need to do this manually if you have an NTP server enabled (which it is by default). Minimum you need to restart, in order, dirsrv.target, krb5kdc, named, httpd, pki-tomcatd. If you restart certmonger it may kick off the renewals for you (or it might not). If you can get the services running back in time then runipa config-show to determine whether this server is configured as the CA renewal server. Only one in the cluster will have this role and the renewals need to take place on that server. If this one isn't it and none of the others report it then you can run: ipa config-mod --ca-renewal-master-server=3D As Flo said the NEED_CSR_GEN_PIN is from your using ipa-cacert-manage. It doesn't affect anything in the short term. rob --===============5128773531211025706==-- From manuel.gugliucci at scaiitec.it Fri Feb 12 15:42:49 2021 Content-Type: multipart/mixed; boundary="===============7830807065629117288==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Fri, 12 Feb 2021 15:42:39 +0000 Message-ID: <20210212154239.21274.63116@mailman01.iad2.fedoraproject.org> In-Reply-To: 4d3a716a-4dae-6e55-a450-833da4daf128@redhat.com --===============7830807065629117288== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Rob, do I have to stop all the IPA services before i move back the date? Now I'm= only moving back date and restarting certmonger. pki-tomcatd is failed so i can't stop/restart it --===============7830807065629117288==-- From rcritten at redhat.com Fri Feb 12 16:05:04 2021 Content-Type: multipart/mixed; boundary="===============3017406877216900466==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Fri, 12 Feb 2021 11:04:49 -0500 Message-ID: In-Reply-To: 20210212154239.21274.63116@mailman01.iad2.fedoraproject.org --===============3017406877216900466== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Manuel Gujo via FreeIPA-users wrote: > Hi Rob, > = > do I have to stop all the IPA services before i move back the date? Now I= 'm only moving back date and restarting certmonger. It wouldn't hurt. You absolutely need to restart things in the past because they can't run in current time with expired certs. > pki-tomcatd is failed so i can't stop/restart it Then go back in the past and we can try to figure out why it won't start then. It won't start now due to expired certs. You can't renew the certs without a working CA. rob --===============3017406877216900466==-- From manuel.gugliucci at scaiitec.it Fri Feb 12 16:11:30 2021 Content-Type: multipart/mixed; boundary="===============0474500137371675838==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Fri, 12 Feb 2021 16:11:21 +0000 Message-ID: <20210212161121.30811.95648@mailman01.iad2.fedoraproject.org> In-Reply-To: c234d6e7-cdf0-2d3d-7bef-da1b01bebc2b@redhat.com --===============0474500137371675838== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable I moved the date before the expiring and restarted the services one by one = as you listed (systemctl restart dirsrv(a)my-domain, systemctl restart krb5= kdc etc.) then: [root(a)ipa1 ~]# ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: STOPPED (if I do systemctl status named it says running) httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: RUNNING pki-tomcatd Service: STOPPED ipa-otpd Service: STOPPED ipa-dnskeysyncd Service: STOPPED ipa: INFO: The ipactl command was successful pki-tomcatd failed to start: # systemctl restart pki-tomcatd(a)ITEC-LAB Job for pki-tomcatd(a)ITEC-LAB.service failed because the control process e= xited with error code. See "systemctl status pki-tomcatd(a)ITEC-LAB.service= " and "journalctl -xe" for details. # journalctl -xe = nov 17 18:22:31 ipa1.itec.lab systemd[1]: Unit pki-tomcatd(a)ITEC-LAB.servi= ce entered failed state. nov 17 18:22:31 ipa1.itec.lab audispd[24456]: node=3Dipa1.itec.lab type=3DS= ERVICE_START msg=3Daudit(1605637351.916:7091): pid=3D1 uid=3D0 auid=3D42949= 67295 ses=3D4294967295 msg=3D'unit=3Dpki-tomc nov 17 18:22:31 ipa1.itec.lab systemd[1]: pki-tomcatd(a)ITEC-LAB.service fa= iled. nov 17 18:22:31 ipa1.itec.lab polkitd[30556]: Unregistered Authentication A= gent for unix-process:17970:71502359 (system bus name :1.1719, object path = /org/freedesktop/PolicyKit1 --===============0474500137371675838==-- From rcritten at redhat.com Fri Feb 12 16:40:01 2021 Content-Type: multipart/mixed; boundary="===============9024709199035763294==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Fri, 12 Feb 2021 11:39:47 -0500 Message-ID: <3d142af9-aa1a-01c6-0864-013f4f0b2832@redhat.com> In-Reply-To: 20210212161121.30811.95648@mailman01.iad2.fedoraproject.org --===============9024709199035763294== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Manuel Gujo via FreeIPA-users wrote: > I moved the date before the expiring and restarted the services one by on= e as you listed (systemctl restart dirsrv(a)my-domain, systemctl restart kr= b5kdc etc.) > = > then: > [root(a)ipa1 ~]# ipactl status > Directory Service: RUNNING > krb5kdc Service: RUNNING > kadmin Service: RUNNING > named Service: STOPPED (if I do systemctl status named it says running) > httpd Service: RUNNING > ipa-custodia Service: RUNNING > ntpd Service: RUNNING > pki-tomcatd Service: STOPPED > ipa-otpd Service: STOPPED > ipa-dnskeysyncd Service: STOPPED > ipa: INFO: The ipactl command was successful > = > pki-tomcatd failed to start: > = > # systemctl restart pki-tomcatd(a)ITEC-LAB > Job for pki-tomcatd(a)ITEC-LAB.service failed because the control process= exited with error code. See "systemctl status pki-tomcatd(a)ITEC-LAB.servi= ce" and "journalctl -xe" for details. > = > # journalctl -xe = > nov 17 18:22:31 ipa1.itec.lab systemd[1]: Unit pki-tomcatd(a)ITEC-LAB.ser= vice entered failed state. > nov 17 18:22:31 ipa1.itec.lab audispd[24456]: node=3Dipa1.itec.lab type= =3DSERVICE_START msg=3Daudit(1605637351.916:7091): pid=3D1 uid=3D0 auid=3D4= 294967295 ses=3D4294967295 msg=3D'unit=3Dpki-tomc > nov 17 18:22:31 ipa1.itec.lab systemd[1]: pki-tomcatd(a)ITEC-LAB.service = failed. > nov 17 18:22:31 ipa1.itec.lab polkitd[30556]: Unregistered Authentication= Agent for unix-process:17970:71502359 (system bus name :1.1719, object pat= h /org/freedesktop/PolicyKit1 Look in /var/log/pki/pki-tomcat/ca/debug. Find where it tries to start the service and go down from there. Looking at the end of the log is almost always fruitless. rob --===============9024709199035763294==-- From manuel.gugliucci at scaiitec.it Fri Feb 12 18:08:03 2021 Content-Type: multipart/mixed; boundary="===============3252552859192356450==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Fri, 12 Feb 2021 18:07:51 +0000 Message-ID: <20210212180751.11429.12894@mailman01.iad2.fedoraproject.org> In-Reply-To: 3d142af9-aa1a-01c6-0864-013f4f0b2832@redhat.com --===============3252552859192356450== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Here's what I found in /var/log/pki/pki-tomcat/ca/debug Could not connect to LDAP server host ipa1.itec.lab port 636 Error netscape= .ldap.LDAPException: Unable to create socket: java.net.ConnectException: Co= nnection refused (Connection refused) (-1) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapB= oundConnFactory.java:205) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFa= ctory.java:166) at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFa= ctory.java:130) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568) at com.netscape.certsrv.apps.CMS.init(CMS.java:191) at com.netscape.certsrv.apps.CMS.start(CMS.java:1458) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java= :117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja= va:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso= rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.ja= va:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.ja= va:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.ja= va:1218) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.ja= va:1174) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.= java:5377) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.= java:5669) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.j= ava:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:13= 3) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(Container= Base.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(Container= Base.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java= :679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.= java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja= va:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j= ava:624) at java.lang.Thread.run(Thread.java:748) Internal Database Error encountered: Could not connect to LDAP server host = ipa1.itec.lab port 636 Error netscape.ldap.LDAPException: Unable to create = socket: java.net.ConnectException: Connection refused (Connection refused) = (-1) at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689) at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1056) at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:962) at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:568) at com.netscape.certsrv.apps.CMS.init(CMS.java:191) at com.netscape.certsrv.apps.CMS.start(CMS.java:1458) at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java= :117) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja= va:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso= rImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288) at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAsPrivileged(Subject.java:549) at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.ja= va:175) at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.ja= va:124) at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.ja= va:1218) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.ja= va:1174) at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066) at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.= java:5377) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.= java:5669) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.j= ava:899) at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:13= 3) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(Container= Base.java:156) at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(Container= Base.java:145) at java.security.AccessController.doPrivileged(Native Method) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java= :679) at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.= java:1966) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja= va:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j= ava:624) at java.lang.Thread.run(Thread.java:748) [31/Dec/2019:19:55:52][localhost-startStop-1]: CMS.start(): shutdown server [31/Dec/2019:19:55:52][localhost-startStop-1]: CMSEngine.shutdown() --===============3252552859192356450==-- From rcritten at redhat.com Fri Feb 12 18:23:35 2021 Content-Type: multipart/mixed; boundary="===============5455666091608152997==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Fri, 12 Feb 2021 13:23:20 -0500 Message-ID: In-Reply-To: 20210212180751.11429.12894@mailman01.iad2.fedoraproject.org --===============5455666091608152997== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Manuel Gujo via FreeIPA-users wrote: > Here's what I found in /var/log/pki/pki-tomcat/ca/debug > = > Could not connect to LDAP server host ipa1.itec.lab port 636 Error netsca= pe.ldap.LDAPException: Unable to create socket: java.net.ConnectException: = Connection refused (Connection refused) (-1) Double-check that dirsrv is started and listening on port 636. I assume the host name is correct. rob --===============5455666091608152997==-- From manuel.gugliucci at scaiitec.it Mon Feb 15 08:33:47 2021 Content-Type: multipart/mixed; boundary="===============2415229631649351648==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Mon, 15 Feb 2021 08:33:37 +0000 Message-ID: <20210215083337.28700.99515@mailman01.iad2.fedoraproject.org> In-Reply-To: bd5ef15c-5ff6-399e-b788-205ea1c8ad10@redhat.com --===============2415229631649351648== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable # systemctl status dirsrv(a)ITEC-LAB =E2=97=8F dirsrv(a)ITEC-LAB.service - 389 Directory Server ITEC-LAB. Loaded: loaded (/usr/lib/systemd/system/dirsrv(a).service; enabled; vend= or preset: disabled) Active: active (running) since mar 2020-11-17 18:00:26 UTC; 2 months 28 = days ago Main PID: 15817 (ns-slapd) Status: "slapd started: Ready to process requests" CGroup: /system.slice/system-dirsrv.slice/dirsrv(a)ITEC-LAB.service =E2=94=94=E2=94=8015817 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-= ITEC-LAB -i /var/run/dirsrv/slapd-ITEC-LAB.pid feb 12 17:26:03 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 # netstat -tulpn | grep LISTEN tcp 0 0 0.0.0.0:749 0.0.0.0:* LISTEN = 1222/kadmind = tcp 0 0 0.0.0.0:464 0.0.0.0:* LISTEN = 1222/kadmind = tcp 0 0 192.168.20.3:53 0.0.0.0:* LISTEN = 17818/named = tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN = 17818/named = tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN = 24121/sshd = tcp 0 0 0.0.0.0:88 0.0.0.0:* LISTEN = 15869/krb5kdc = tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN = 17818/named = tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN = 24232/master = tcp6 0 0 :::749 :::* LISTEN = 1222/kadmind = tcp6 0 0 :::80 :::* LISTEN = 16122/httpd = tcp6 0 0 :::464 :::* LISTEN = 1222/kadmind = tcp6 0 0 :::53 :::* LISTEN = 17818/named = tcp6 0 0 :::22 :::* LISTEN = 24121/sshd = tcp6 0 0 :::88 :::* LISTEN = 15869/krb5kdc = tcp6 0 0 ::1:953 :::* LISTEN = 17818/named = tcp6 0 0 ::1:25 :::* LISTEN = 24232/master = tcp6 0 0 :::443 :::* LISTEN = 16122/httpd = dirsrv is started but I don't see port 636 on this list, how can I open it = for dirsrv? --===============2415229631649351648==-- From rcritten at redhat.com Mon Feb 15 14:30:48 2021 Content-Type: multipart/mixed; boundary="===============4483985877845769890==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Mon, 15 Feb 2021 09:30:34 -0500 Message-ID: In-Reply-To: 20210215083337.28700.99515@mailman01.iad2.fedoraproject.org --===============4483985877845769890== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Manuel Gujo via FreeIPA-users wrote: > # systemctl status dirsrv(a)ITEC-LAB > =E2=97=8F dirsrv(a)ITEC-LAB.service - 389 Directory Server ITEC-LAB. > Loaded: loaded (/usr/lib/systemd/system/dirsrv(a).service; enabled; ve= ndor preset: disabled) > Active: active (running) since mar 2020-11-17 18:00:26 UTC; 2 months 2= 8 days ago > Main PID: 15817 (ns-slapd) > Status: "slapd started: Ready to process requests" > CGroup: /system.slice/system-dirsrv.slice/dirsrv(a)ITEC-LAB.service > =E2=94=94=E2=94=8015817 /usr/sbin/ns-slapd -D /etc/dirsrv/slap= d-ITEC-LAB -i /var/run/dirsrv/slapd-ITEC-LAB.pid > = > feb 12 17:26:03 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 1 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 2 > feb 14 03:35:04 ipa1.itec.lab ns-slapd[15817]: GSSAPI server step 3 > = > # netstat -tulpn | grep LISTEN > tcp 0 0 0.0.0.0:749 0.0.0.0:* LISTE= N 1222/kadmind = > tcp 0 0 0.0.0.0:464 0.0.0.0:* LISTE= N 1222/kadmind = > tcp 0 0 192.168.20.3:53 0.0.0.0:* LISTE= N 17818/named = > tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTE= N 17818/named = > tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTE= N 24121/sshd = > tcp 0 0 0.0.0.0:88 0.0.0.0:* LISTE= N 15869/krb5kdc = > tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTE= N 17818/named = > tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTE= N 24232/master = > tcp6 0 0 :::749 :::* LISTE= N 1222/kadmind = > tcp6 0 0 :::80 :::* LISTE= N 16122/httpd = > tcp6 0 0 :::464 :::* LISTE= N 1222/kadmind = > tcp6 0 0 :::53 :::* LISTE= N 17818/named = > tcp6 0 0 :::22 :::* LISTE= N 24121/sshd = > tcp6 0 0 :::88 :::* LISTE= N 15869/krb5kdc = > tcp6 0 0 ::1:953 :::* LISTE= N 17818/named = > tcp6 0 0 ::1:25 :::* LISTE= N 24232/master = > tcp6 0 0 :::443 :::* LISTE= N 16122/httpd = > = > dirsrv is started but I don't see port 636 on this list, how can I open i= t for dirsrv? In fact it isn't listening on any port. Stop dirsrv and look in /etc/dirsrv/slapd-ITEC-LAB/dse.ldif for nsslapd-port which should be 389 and nsslapd-security which should be on. If not then fix it and restart dirsrv. That should fix it. rob --===============4483985877845769890==-- From manuel.gugliucci at scaiitec.it Mon Feb 15 15:38:29 2021 Content-Type: multipart/mixed; boundary="===============5827145078568548144==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Mon, 15 Feb 2021 15:38:18 +0000 Message-ID: <20210215153818.11220.71701@mailman01.iad2.fedoraproject.org> In-Reply-To: d2fcd4d5-42d7-b1a0-20e9-390c414479e2@redhat.com --===============5827145078568548144== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Rob, so in "/etc/dirsrv/slapd-ITEC-LAB/dse.ldif", nsslapd-port was '0' and nssla= pd-security was off, I fixed it and now it's listening on port 389 and 636: # netstat -tulpn | grep LISTEN | grep ns-slapd tcp6 0 0 :::636 :::* LISTEN = 30606/ns-slapd = tcp6 0 0 :::389 :::* LISTEN = 30606/ns-slapd = Then I tried to restart all the ipactl services one by one. pki-tomcatd kee= ps failing and /var/log/pki/pki-tomcat/ca/debug does not log anymore (last = log is the one i sent you above, 31 Dec 2019) I resubmitted all the expired certs and restarting cermonger but certs keep= being unreachable. from certmonger logs: nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: Forw= arding request to dogtag-ipa-renew-agent nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: GET htt= p://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit?profileId=3DcaServerCert&seri= al_num=3D20&renewal=3Dtrue&xml=3Dtrue& nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: (null) nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: dogt= ag-ipa-renew-agent returned 3 nov 17 18:11:47 ipa1.itec.lab certmonger[30685]: 2020-11-17 18:11:47 [30685= ] Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit: C= ouldn't connect to server. in certmonger's log I also saw these: nov 17 18:11:01 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30741]: Trac= eback (most recent call last): Fi= le "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, in= = sys.exit(main()) Fi= le "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 495, in= main = api.finalize() Fi= le "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 740, in fina= lize = self.__do_if_not_done('load_plugins') Fi= le "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 431, in __do= _if_not_done = getattr(self, name)() Fi= le "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 620, in load= _plugins = self.add_package(package) Fi= le "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 647, in add_= package = module =3D importlib.import_module(name) Fi= le "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module = __import__(name) Fi= le "/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 32,= in = from ipaserver.install import bindinstance, dnskeysyncinstance Fi= le "/usr/lib/python2.7/site-packages/ipaserver/install/dnskeysyncinstance.p= y", line 17, in = from ipaserver import p11helper as _ipap11helper Fi= le "/usr/lib/python2.7/site-packages/ipaserver/p11helper.py", line 342, in = = _libp11_kit =3D _ffi.dlopen(ctypes.util.find_library('p11-kit')) Fi= le "/usr/lib64/python2.7/ctypes/util.py", line 244, in find_library = return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) Fi= le "/usr/lib64/python2.7/ctypes/util.py", line 233, in _findSoname_ldconfig = f =3D os.popen('/sbin/ldconfig -p 2>/dev/null') OSEr= ror: [Errno 12] Cannot allocate memory --===============5827145078568548144==-- From rcritten at redhat.com Mon Feb 15 20:02:21 2021 Content-Type: multipart/mixed; boundary="===============8024469623496485149==" MIME-Version: 1.0 From: Rob Crittenden To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Mon, 15 Feb 2021 15:02:07 -0500 Message-ID: <97a36513-8c3b-23fa-bff6-5a6a9a4c06fc@redhat.com> In-Reply-To: 20210215153818.11220.71701@mailman01.iad2.fedoraproject.org --===============8024469623496485149== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Manuel Gujo via FreeIPA-users wrote: > Hi Rob, > = > so in "/etc/dirsrv/slapd-ITEC-LAB/dse.ldif", nsslapd-port was '0' and nss= lapd-security was off, I fixed it and now it's listening on port 389 and 63= 6: > = > # netstat -tulpn | grep LISTEN | grep ns-slapd > tcp6 0 0 :::636 :::* LISTE= N 30606/ns-slapd = > tcp6 0 0 :::389 :::* LISTE= N 30606/ns-slapd = > = > Then I tried to restart all the ipactl services one by one. pki-tomcatd k= eeps failing and /var/log/pki/pki-tomcat/ca/debug does not log anymore (las= t log is the one i sent you above, 31 Dec 2019) > = > I resubmitted all the expired certs and restarting cermonger but certs ke= ep being unreachable. If the CA isn't running then there is no point in resubmitting the certmonger requests. It is guaranteed to fail with UNREACHABLE. Check the journalctl output and the other logs, like catalina, in /var/log/pki/pki-tomcat for more information on why it failed to start. > = > from certmonger logs: > = > nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: Fo= rwarding request to dogtag-ipa-renew-agent > nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: GET h= ttp://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit?profileId=3DcaServerCert&se= rial_num=3D20&renewal=3Dtrue&xml=3Dtrue& > nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-renew-agent-submit[31183]: (null) > nov 17 18:11:47 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30764]: do= gtag-ipa-renew-agent returned 3 > nov 17 18:11:47 ipa1.itec.lab certmonger[30685]: 2020-11-17 18:11:47 [306= 85] Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profileSubmit:= Couldn't connect to server. > = > in certmonger's log I also saw these: > = > nov 17 18:11:01 ipa1.itec.lab dogtag-ipa-ca-renew-agent-submit[30741]: Tr= aceback (most recent call last): > = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 533, = in > = sys.exit(main()) > = File "/usr/libexec/certmonger/dogtag-ipa-ca-renew-agent-submit", line 495, = in main > = api.finalize() > = File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 740, in fi= nalize > = self.__do_if_not_done('load_plugins') > = File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 431, in __= do_if_not_done > = getattr(self, name)() > = File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 620, in lo= ad_plugins > = self.add_package(package) > = File "/usr/lib/python2.7/site-packages/ipalib/plugable.py", line 647, in ad= d_package > = module =3D importlib.import_module(name) > = File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module > = __import__(name) > = File "/usr/lib/python2.7/site-packages/ipaserver/plugins/server.py", line 3= 2, in > = from ipaserver.install import bindinstance, dnskeysyncinstance > = File "/usr/lib/python2.7/site-packages/ipaserver/install/dnskeysyncinstance= .py", line 17, in > = from ipaserver import p11helper as _ipap11helper > = File "/usr/lib/python2.7/site-packages/ipaserver/p11helper.py", line 342, i= n > = _libp11_kit =3D _ffi.dlopen(ctypes.util.find_library('p11-kit')) > = File "/usr/lib64/python2.7/ctypes/util.py", line 244, in find_library > = return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name)) > = File "/usr/lib64/python2.7/ctypes/util.py", line 233, in _findSoname_ldconf= ig > = f =3D os.popen('/sbin/ldconfig -p 2>/dev/null') > OS= Error: [Errno 12] Cannot allocate memory Is this host memory-constrained? How much RAM does it have? rob --===============8024469623496485149==-- From manuel.gugliucci at scaiitec.it Tue Feb 16 08:58:22 2021 Content-Type: multipart/mixed; boundary="===============0353205060416642309==" MIME-Version: 1.0 From: Manuel Gujo To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: IPA certs expired, pki-tomcatd fails to start Date: Tue, 16 Feb 2021 08:58:13 +0000 Message-ID: <20210216085813.14637.6920@mailman01.iad2.fedoraproject.org> In-Reply-To: 97a36513-8c3b-23fa-bff6-5a6a9a4c06fc@redhat.com --===============0353205060416642309== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable > Manuel Gujo via FreeIPA-users wrote: > = > If the CA isn't running then there is no point in resubmitting the > certmonger requests. It is guaranteed to fail with UNREACHABLE. > = > Check the journalctl output and the other logs, like catalina, in > /var/log/pki/pki-tomcat for more information on why it failed to start. > = > = > = > Is this host memory-constrained? How much RAM does it have? > = > rob there's new log on debug. Catalina does not log anything (0kb per file). in debug: Could not connect to LDAP server host ipa1.itec.lab port 636 Error netscape= .ldap.LDAPException: Unable to create socket: org.mozilla.jss.ssl.SSLSocket= Exception: org.mozilla.jss.ssl.SSLSocketExc eption: SSL_ForceHandshake failed: (-8181) Peer's Certificate has expired. = (-1) in "system" logs says the same thing of debugs' When I try to run 'ipactl start' without -f option, it says this: # ipactl start IPA version error: data needs to be upgraded (expected version '4.6.8-5.el7= .centos', current version '4.4.0-14.el7.centos.4') then after a while it fails and in /var/log/ipaupgrade.log says: 2020-11-17T18:25:05Z DEBUG httplib request failed: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 220, in= _httplib_request conn.request(method, path, body=3Drequest_body, headers=3Dheaders) File "/usr/lib64/python2.7/httplib.py", line 1056, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1090, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1052, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 890, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 852, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 1266, in connect HTTPConnection.connect(self) File "/usr/lib64/python2.7/httplib.py", line 833, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused 2020-11-17T18:25:05Z ERROR IPA server upgrade failed: Inspect /var/log/ipau= pgrade.log and run command ipa-server-upgrade manually. 2020-11-17T18:25:05Z DEBUG File "/usr/lib/python2.7/site-packages/ipapyth= on/admintool.py", line 178, in execute return_value =3D self.run() File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgra= de.py", line 54, in run server.upgrade() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.p= y", line 2176, in upgrade upgrade_configuration() File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.p= y", line 2059, in upgrade_configuration cainstance.repair_profile_caIPAserviceCert() File "/usr/lib/python2.7/site-packages/ipaserver/install/cainstance.py", = line 1949, in repair_profile_caIPAserviceCert with api.Backend.ra_certprofile as profile_api: File "/usr/lib/python2.7/site-packages/ipaserver/plugins/dogtag.py", line= 1311, in __enter__ method=3D'GET' File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 167, in= https_request method=3Dmethod, headers=3Dheaders) File "/usr/lib/python2.7/site-packages/ipapython/dogtag.py", line 229, in= _httplib_request raise NetworkError(uri=3Duri, error=3Dstr(e)) 2020-11-17T18:25:05Z DEBUG The ipa-server-upgrade command failed, exception= : NetworkError: cannot connect to 'https://ipa1.itec.lab:8443/ca/rest/accou= nt/login': [Errno 111] Connection refused 2020-11-17T18:25:05Z ERROR Unexpected error - see /var/log/ipaupgrade.log f= or details: After this run, I noticed that some of the certs went on Monitoring state # getcert list Number of certificates and requests being tracked: 9. Request ID '20191231201955': status: MONITORING stuck: no key pair storage: type=3DFILE,location=3D'/var/kerberos/krb5kdc/kdc.key' certificate: type=3DFILE,location=3D'/var/kerberos/krb5kdc/kdc.crt' CA: SelfSign issuer: CN=3Dipa1.itec.lab,O=3DITEC.LAB subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB expires: 2022-02-08 15:59:12 UTC principal name: krbtgt/ITEC.LAB(a)ITEC.LAB certificate template/profile: KDCs_PKINIT_Certs pre-save command: = post-save command: /usr/libexec/ipa/certmonger/renew_kdc_cert track: yes auto-renew: yes Request ID '20201117182331': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profile= Submit: Couldn't connect to server. stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'auditSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DCA Audit,O=3DITEC.LAB expires: 2020-12-08 09:35:14 UTC key usage: digitalSignature,nonRepudiation pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigning= Cert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182333': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profile= Submit: Couldn't connect to server. stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'ocspSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DOCSP Subsystem,O=3DITEC.LAB expires: 2020-12-08 09:38:07 UTC eku: id-kp-OCSPSigning pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningC= ert cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182335': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'subsystemCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DCA Subsystem,O=3DITEC.LAB expires: 2022-11-07 18:24:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCer= t cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182336': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'caSigningCert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DCertificate Authority,O=3DITEC.LAB expires: 2037-01-25 14:22:25 UTC key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCer= t cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182338': status: CA_UNREACHABLE ca-error: Error 7 connecting to http://ipa1.itec.lab:8080/ca/ee/ca/profile= Submit: Couldn't connect to server. stuck: no key pair storage: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.key' certificate: type=3DFILE,location=3D'/var/lib/ipa/ra-agent.pem' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3DIPA RA,O=3DITEC.LAB expires: 2020-12-08 09:37:47 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert track: yes auto-renew: yes Request ID '20201117182339': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nick= name=3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB',pin set certificate: type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname= =3D'Server-Cert cert-pki-ca',token=3D'NSS Certificate DB' CA: dogtag-ipa-ca-renew-agent issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB expires: 2022-11-07 18:24:56 UTC key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert = cert-pki-ca" track: yes auto-renew: yes Request ID '20201117182342': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/dirsrv/slapd-ITEC-LAB',nic= kname=3D'Server-Cert',token=3D'NSS Certificate DB',pinfile=3D'/etc/dirsrv/s= lapd-ITEC-LAB/pwdfile.txt' certificate: type=3DNSSDB,location=3D'/etc/dirsrv/slapd-ITEC-LAB',nickname= =3D'Server-Cert',token=3D'NSS Certificate DB' CA: IPA issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB expires: 2020-12-30 09:35:16 UTC principal name: ldap/ipa1.itec.lab(a)ITEC.LAB key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: = post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv ITEC-LAB track: yes auto-renew: yes Request ID '20201117182351': status: MONITORING stuck: no key pair storage: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'S= erver-Cert',token=3D'NSS Certificate DB',pinfile=3D'/etc/httpd/alias/pwdfil= e.txt' certificate: type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server= -Cert',token=3D'NSS Certificate DB' CA: IPA issuer: CN=3DCertificate Authority,O=3DITEC.LAB subject: CN=3Dipa1.itec.lab,O=3DITEC.LAB expires: 2020-12-30 09:35:04 UTC principal name: HTTP/ipa1.itec.lab(a)ITEC.LAB key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment eku: id-kp-serverAuth,id-kp-clientAuth pre-save command: = post-save command: /usr/libexec/ipa/certmonger/restart_httpd track: yes auto-renew: yes # ipactl status Directory Service: RUNNING krb5kdc Service: RUNNING kadmin Service: RUNNING named Service: STOPPED httpd Service: RUNNING ipa-custodia Service: RUNNING ntpd Service: STOPPED pki-tomcatd Service: RUNNING ipa-otpd Service: RUNNING ipa-dnskeysyncd Service: RUNNING ipa: INFO: The ipactl command was successful but pki-tomcatd still fails if I try to restart it and in the debug logs: [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet:service() uri =3D = /ca/admin/ca/getStatus [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet: caGetStatus start= to service. [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: Failed to read product versio= n String. java.io.FileNotFoundException: /usr/share/pki/CS_SERVER_VERSION (= No such file or directory) [17/Nov/2020:18:32:34][http-bio-8080-exec-7]: CMSServlet: curDate=3DTue Nov= 17 18:32:34 UTC 2020 id=3DcaGetStatus time=3D9 IPA VM has 2 CPU and 4GB of RAM, it never goes up to 90% of the usage --===============0353205060416642309==--