From flo at redhat.com Wed Jun 20 16:49:35 2018 Content-Type: multipart/mixed; boundary="===============1358139270359344915==" MIME-Version: 1.0 From: Florence Blanc-Renaud To: freeipa-users at lists.fedorahosted.org Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade doesn't complete, pki-tomcatd won't start Date: Wed, 20 Jun 2018 18:49:04 +0200 Message-ID: In-Reply-To: 9dbd45efa82b4a9c943e1e3bd8071d43@cinia.fi --===============1358139270359344915== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote: > Hello all! > = > I have very similiar problem as this one: > = > https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedora= hosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/ > = > ipa-server-upgrade fails as below > = > -- > = > Update complete > = > Upgrading IPA services > = > Upgrading the configuration of the IPA services > = > [Verifying that root certificate is published] > = > [Migrate CRL publish directory] > = > CRL tree already moved > = > [Verifying that CA proxy configuration is correct] > = > IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run = > command ipa-server-upgrade manually. > = > CA did not start in 300.0s > = > The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for = > more information > = > -- > = > And the log tells me that CA returns status 500 > = > -- > = > DEBUG Waiting for CA to start... > = > DEBUG request POST http://<>:8080/ca/admin/ca/getStatus = > > = > DEBUG request body '' > = > DEBUG response status 500 > = > DEBUG response headers Server: Apache-Coyote/1.1 > = > Content-Type: text/html;charset=3Dutf-8 > = > Content-Language: en > = > Content-Length: 2208 > = > Date: Fri, 15 Jun 2018 10:05:29 GMT > = > Connection: close > = > DEBUG response body 'Apache Tomcat/7.0.76 - Error = > report = >

HTTP Status 500 - Subsystem unavailable


size=3D"1" noshade=3D"noshade">

type Exception = > report

message Subsystem = > unavailable

description The server encountered an = > internal error that prevented it from fulfilling this = > request.

exception = >

javax.ws.rs.ServiceUnavailableException: Subsystem =

> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints=
(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBas=
e.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorRep=
ortValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.Ac=
cessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connect=
or.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http1=
1.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg=
.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractP=
rotocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor=
.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWo=
rker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecut=
or$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.thread=
s.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.r=
un(Thread.java:748)\n

note = > The full stack trace of the root cause is available in the Apache = > Tomcat/7.0.76 logs.


Apache = > Tomcat/7.0.76

' > = > DEBUG The CA status is: check interrupted due to error: Retrieving CA = > status failed with status 500 > = > DEBUG Waiting for CA to start... > = > ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run = > command ipa-server-upgrade manually. > = > File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line = > 172, in execute > = > =C2=A0=C2=A0=C2=A0 return_value =3D self.run() > = > =C2=A0 File = > "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py= ", = > line 48, in run > = > =C2=A0=C2=A0=C2=A0 raise admintool.ScriptError(str(e)) > = > The ipa-server-upgrade command failed, exception: ScriptError: CA did = > not start in 300.0s > = > ERROR CA did not start in 300.0s > = > -- > = > With command =E2=80=9Cipactl start --ignore-service-failures=E2=80=9D I c= an start all = > the services but pki-tomcatd. > = > -- > = > Directory Service: RUNNING > = > krb5kdc Service: RUNNING > = > kadmin Service: RUNNING > = > named Service: RUNNING > = > httpd Service: RUNNING > = > pki-tomcatd Service: STOPPED > = > smb Service: RUNNING > = > winbind Service: RUNNING > = > ipa-otpd Service: RUNNING > = > ipa-dnskeysyncd Service: RUNNING > = > ipa: INFO: The ipactl command was successful > = > -- > = > Suggested resolution to above problem doesn=E2=80=99t help me since the L= DAP and = > NSS DB seem to have same certificates (some difference in wrapping but = > the string is same if I take out the line breaks) and even the serial = > number matches. > = > -- > = > certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a > = > -----BEGIN CERTIFICATE----- > = > MIIDjD=E2=80=A6 > = > =E2=80=A6Prh2G > = > -----END CERTIFICATE----- > = > certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' = > |grep Serial > = > =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Serial Number: 4 (0x4) > = > ldapsearch -LLL -D 'cn=3Ddirectory manager' -W -b = > uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca userCertificate description seeAlso > = > Enter LDAP Password: > = > dn: uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca > = > userCertificate:: MIIDjD=E2=80=A6 > = > =E2=80=A6Prh2 > = > G > = > description: 2;4;CN=3DCertificate Authority,O=3D<>;CN=3DCA Subsyst= em, > = > O=3D<> > = > seeAlso: CN=3DCA Subsystem,O=3D<> > = > -- > = > And here=E2=80=99s where my actual knowledge of things end. I=E2=80=99ve = been trying to = > figure out all kind of logs (tomcat, Kerberos, directory server, =E2=80= =A6) but = > haven=E2=80=99t found a solid reason for it. I=E2=80=99m starting to beli= eve this is a = > certificate issue, because although =E2=80=9Cgetcert list=E2=80=9D tells = me that the = > certificate status is =E2=80=9CMonitoring=E2=80=9D on all certificates th= e expiry date = > is already in the past (current date 20.6.2018, certificate expiry = > 21.03.2018) on 4 certificates and it won=E2=80=99t update even if I resub= mit it = > or delete certificate and manually redo it (it got the same date as the = > =E2=80=9Cold ones=E2=80=9D). The =E2=80=9Cmain certs=E2=80=9D (=E2=80=9Cc= aSigningCert cert-pki-ca=E2=80=9D, =E2=80=9CServer-Cert = > cert-pki-ca=E2=80=9D and two directory server certs) are valid for years = (until = > 2020+). > = > -- > = > Request ID '20160331084234': > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni= ngCert cert-pki-ca',token=3D'NSS = > Certificate DB',pin set > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni= ngCert cert-pki-ca',token=3D'NSS = > Certificate DB' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: dogtag-ipa-ca-renew-agent > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Auth= ority,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3DOCSP Subsystem,= O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2018-03-21 09:42:04 = UTC > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: digitalSignature,n= onRepudiation,keyCertSign,cRLSign > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-OCSPSigning > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: /usr/libexe= c/ipa/certmonger/stop_pkicad > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/libex= ec/ipa/certmonger/renew_ca_cert = > "ocspSigningCert cert-pki-ca" > = > =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0track: yes > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes > = > Request ID '20160331085008': > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: = > type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',token= =3D'NSS = > Certificate DB',pinfile=3D'/etc/httpd/alias/pwdfile.txt' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: = > type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',token= =3D'NSS = > Certificate DB' > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: IPA > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Auth= ority,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3D<= >,O=3D<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2020-03-04 09:58:23 = UTC > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 principal name: HTTP/<>@<> > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: = > digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-serverAuth,id-kp-c= lientAuth > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/lib64= /ipa/certmonger/restart_httpd > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 track: yes > = > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes > = > -- > = > Has anyone else bumped into same kind of issues? Any ideas where I = > should continue looking? I=E2=80=99m starting to run out of ideas=E2=80= =A6 > = > Eemeli Jokinen > = > = > = > _______________________________________________ > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org > To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.= org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user= s(a)lists.fedorahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/ > = Hi, does your topology include multiple CA instances? You need first to find = which master is the CA renewal master: ipa config-show | grep "renewal master" On this host, check that the certificates are still valid and consistent = with the content of the LDAP entries. If it is not the case, you need to = repair the CA renewal master first. When the CA renewal master is OK, check if the replication is working = with the other CA instances, and repair the other masters. HTH, Flo --===============1358139270359344915==--