From flo at redhat.com Wed Jun 20 16:49:35 2018
Content-Type: multipart/mixed; boundary="===============1358139270359344915=="
MIME-Version: 1.0
From: Florence Blanc-Renaud
To: freeipa-users at lists.fedorahosted.org
Subject: [Freeipa-users] Re: Problems after IPA upgrade: ipa-server-upgrade
doesn't complete, pki-tomcatd won't start
Date: Wed, 20 Jun 2018 18:49:04 +0200
Message-ID:
In-Reply-To: 9dbd45efa82b4a9c943e1e3bd8071d43@cinia.fi
--===============1358139270359344915==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
> Hello all!
> =
> I have very similiar problem as this one:
> =
> https://lists.fedorahosted.org/archives/list/freeipa-users(a)lists.fedora=
hosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
> =
> ipa-server-upgrade fails as below
> =
> --
> =
> Update complete
> =
> Upgrading IPA services
> =
> Upgrading the configuration of the IPA services
> =
> [Verifying that root certificate is published]
> =
> [Migrate CRL publish directory]
> =
> CRL tree already moved
> =
> [Verifying that CA proxy configuration is correct]
> =
> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run =
> command ipa-server-upgrade manually.
> =
> CA did not start in 300.0s
> =
> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for =
> more information
> =
> --
> =
> And the log tells me that CA returns status 500
> =
> --
> =
> DEBUG Waiting for CA to start...
> =
> DEBUG request POST http://<>:8080/ca/admin/ca/getStatus =
>
> =
> DEBUG request body ''
> =
> DEBUG response status 500
> =
> DEBUG response headers Server: Apache-Coyote/1.1
> =
> Content-Type: text/html;charset=3Dutf-8
> =
> Content-Language: en
> =
> Content-Length: 2208
> =
> Date: Fri, 15 Jun 2018 10:05:29 GMT
> =
> Connection: close
> =
> DEBUG response body 'Apache Tomcat/7.0.76 - Error =
> report =
> HTTP Status 500 - Subsystem unavailable
size=3D"1" noshade=3D"noshade">type Exception =
> report
message Subsystem =
> unavailable
description The server encountered an =
> internal error that prevented it from fulfilling this =
> request.
exception =
>
javax.ws.rs.ServiceUnavailableException: Subsystem =
> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints=
(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBas=
e.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.valves.ErrorRep=
ortValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.Ac=
cessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connect=
or.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http1=
1.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)\n\torg=
.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractP=
rotocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor=
.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWo=
rker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecut=
or$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.thread=
s.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.r=
un(Thread.java:748)\n
note =
> The full stack trace of the root cause is available in the Apache =
> Tomcat/7.0.76 logs.
Apache =
> Tomcat/7.0.76
'
> =
> DEBUG The CA status is: check interrupted due to error: Retrieving CA =
> status failed with status 500
> =
> DEBUG Waiting for CA to start...
> =
> ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run =
> command ipa-server-upgrade manually.
> =
> File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line =
> 172, in execute
> =
> =C2=A0=C2=A0=C2=A0 return_value =3D self.run()
> =
> =C2=A0 File =
> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py=
", =
> line 48, in run
> =
> =C2=A0=C2=A0=C2=A0 raise admintool.ScriptError(str(e))
> =
> The ipa-server-upgrade command failed, exception: ScriptError: CA did =
> not start in 300.0s
> =
> ERROR CA did not start in 300.0s
> =
> --
> =
> With command =E2=80=9Cipactl start --ignore-service-failures=E2=80=9D I c=
an start all =
> the services but pki-tomcatd.
> =
> --
> =
> Directory Service: RUNNING
> =
> krb5kdc Service: RUNNING
> =
> kadmin Service: RUNNING
> =
> named Service: RUNNING
> =
> httpd Service: RUNNING
> =
> pki-tomcatd Service: STOPPED
> =
> smb Service: RUNNING
> =
> winbind Service: RUNNING
> =
> ipa-otpd Service: RUNNING
> =
> ipa-dnskeysyncd Service: RUNNING
> =
> ipa: INFO: The ipactl command was successful
> =
> --
> =
> Suggested resolution to above problem doesn=E2=80=99t help me since the L=
DAP and =
> NSS DB seem to have same certificates (some difference in wrapping but =
> the string is same if I take out the line breaks) and even the serial =
> number matches.
> =
> --
> =
> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' -a
> =
> -----BEGIN CERTIFICATE-----
> =
> MIIDjD=E2=80=A6
> =
> =E2=80=A6Prh2G
> =
> -----END CERTIFICATE-----
> =
> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca' =
> |grep Serial
> =
> =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Serial Number: 4 (0x4)
> =
> ldapsearch -LLL -D 'cn=3Ddirectory manager' -W -b =
> uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca userCertificate description seeAlso
> =
> Enter LDAP Password:
> =
> dn: uid=3Dpkidbuser,ou=3Dpeople,o=3Dipaca
> =
> userCertificate:: MIIDjD=E2=80=A6
> =
> =E2=80=A6Prh2
> =
> G
> =
> description: 2;4;CN=3DCertificate Authority,O=3D<>;CN=3DCA Subsyst=
em,
> =
> O=3D<>
> =
> seeAlso: CN=3DCA Subsystem,O=3D<>
> =
> --
> =
> And here=E2=80=99s where my actual knowledge of things end. I=E2=80=99ve =
been trying to =
> figure out all kind of logs (tomcat, Kerberos, directory server, =E2=80=
=A6) but =
> haven=E2=80=99t found a solid reason for it. I=E2=80=99m starting to beli=
eve this is a =
> certificate issue, because although =E2=80=9Cgetcert list=E2=80=9D tells =
me that the =
> certificate status is =E2=80=9CMonitoring=E2=80=9D on all certificates th=
e expiry date =
> is already in the past (current date 20.6.2018, certificate expiry =
> 21.03.2018) on 4 certificates and it won=E2=80=99t update even if I resub=
mit it =
> or delete certificate and manually redo it (it got the same date as the =
> =E2=80=9Cold ones=E2=80=9D). The =E2=80=9Cmain certs=E2=80=9D (=E2=80=9Cc=
aSigningCert cert-pki-ca=E2=80=9D, =E2=80=9CServer-Cert =
> cert-pki-ca=E2=80=9D and two directory server certs) are valid for years =
(until =
> 2020+).
> =
> --
> =
> Request ID '20160331084234':
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: =
> type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni=
ngCert cert-pki-ca',token=3D'NSS =
> Certificate DB',pin set
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: =
> type=3DNSSDB,location=3D'/etc/pki/pki-tomcat/alias',nickname=3D'ocspSigni=
ngCert cert-pki-ca',token=3D'NSS =
> Certificate DB'
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: dogtag-ipa-ca-renew-agent
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Auth=
ority,O=3D<>
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3DOCSP Subsystem,=
O=3D<>
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2018-03-21 09:42:04 =
UTC
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: digitalSignature,n=
onRepudiation,keyCertSign,cRLSign
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-OCSPSigning
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command: /usr/libexe=
c/ipa/certmonger/stop_pkicad
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/libex=
ec/ipa/certmonger/renew_ca_cert =
> "ocspSigningCert cert-pki-ca"
> =
> =C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0track: yes
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes
> =
> Request ID '20160331085008':
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: MONITORING
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 stuck: no
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key pair storage: =
> type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',token=
=3D'NSS =
> Certificate DB',pinfile=3D'/etc/httpd/alias/pwdfile.txt'
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 certificate: =
> type=3DNSSDB,location=3D'/etc/httpd/alias',nickname=3D'Server-Cert',token=
=3D'NSS =
> Certificate DB'
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CA: IPA
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 issuer: CN=3DCertificate Auth=
ority,O=3D<>
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 subject: CN=3D<=
>,O=3D<>
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 expires: 2020-03-04 09:58:23 =
UTC
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 principal name: HTTP/<>@<>
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 key usage: =
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 eku: id-kp-serverAuth,id-kp-c=
lientAuth
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pre-save command:
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 post-save command: /usr/lib64=
/ipa/certmonger/restart_httpd
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 track: yes
> =
> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 auto-renew: yes
> =
> --
> =
> Has anyone else bumped into same kind of issues? Any ideas where I =
> should continue looking? I=E2=80=99m starting to run out of ideas=E2=80=
=A6
> =
> Eemeli Jokinen
> =
> =
> =
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.=
org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/freeipa-user=
s(a)lists.fedorahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
> =
Hi,
does your topology include multiple CA instances? You need first to find =
which master is the CA renewal master:
ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent =
with the content of the LDAP entries. If it is not the case, you need to =
repair the CA renewal master first.
When the CA renewal master is OK, check if the replication is working =
with the other CA instances, and repair the other masters.
HTH,
Flo
--===============1358139270359344915==--