Hi!
We do have 2 IPA nodes configured but the second node has been down for some time. Tried
to update it to same version as node1:
- Won't start tells me to use ipa-server-upgrade
- Ipa-server-upgrade fails at start, doesn't start directory server
- /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse errors but
last row tells me
--
EMERG - main - Fatal Error---No ports specified. Exiting now.
--
On node1 ipa config-show results only
--
ipa: ERROR: did not receive Kerberos credentials
--
You need to run kinit to get kerberos credentials before any ipa *
command. If the ipa stack is stopped because pki-tomcat refused to
start, you can force the start up, ignoring failed services:
$ sudo ipactl start --ignore-service-failures
Then to check who is renewal master:
$ sudo kinit admin
$ sudo ipa config-show
If node1 is not the renewal master, make it the renewal master from now
on (follow instructions from
).
After this step, check the certificates in the NSS databases vs the
certificates stored in LDAP.
HTH,
Flo
Eemeli
-----Original Message-----
From: Florence Blanc-Renaud [mailto:flo@redhat.com]
Sent: keskiviikko 20. kesäkuuta 2018 19.49
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
Cc: Jokinen Eemeli <Eemeli.Jokinen(a)cinia.fi>
Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade doesn't
complete, pki-tomcatd won't start
On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
> Hello all!
>
> I have very similiar problem as this one:
>
>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedor
>
ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
>
> ipa-server-upgrade fails as below
>
> --
>
> Update complete
>
> Upgrading IPA services
>
> Upgrading the configuration of the IPA services
>
> [Verifying that root certificate is published]
>
> [Migrate CRL publish directory]
>
> CRL tree already moved
>
> [Verifying that CA proxy configuration is correct]
>
> IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
> command ipa-server-upgrade manually.
>
> CA did not start in 300.0s
>
> The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
> more information
>
> --
>
> And the log tells me that CA returns status 500
>
> --
>
> DEBUG Waiting for CA to start...
>
> DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus
> <http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus>
>
> DEBUG request body ''
>
> DEBUG response status 500
>
> DEBUG response headers Server: Apache-Coyote/1.1
>
> Content-Type: text/html;charset=utf-8
>
> Content-Language: en
>
> Content-Length: 2208
>
> Date: Fri, 15 Jun 2018 10:05:29 GMT
>
> Connection: close
>
> DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 -
Error
> report</title><style><!--H1
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
> D76;font-size:22px;}
> H2
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
> D76;font-size:16px;}
> H3
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
> D76;font-size:14px;}
> BODY
> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:whit
> e;} B
> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
> D76;}
> P
> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font
> -size:12px;}A {color : black;}A.name {color : black;}HR {color :
> #525D76;}--></style> </head><body><h1>HTTP Status 500 -
Subsystem
> unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b>
> Exception report</p><p><b>message</b> <u>Subsystem
> unavailable</u></p><p><b>description</b> <u>The
server encountered an
> internal error that prevented it from fulfilling this
> request.</u></p><p><b>exception</b>
> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem
> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstrai
> nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authenti
> catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.va
> lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.
> catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.
> apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445
> )\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
> Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abstr
> actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.
> tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\
> n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
> r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Thr
> eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread
> $WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre
> ad.java:748)\n</pre></p><p><b>note</b>
> <u>The full stack trace of the root cause is available in the Apache
> Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
> Tomcat/7.0.76</h3></body></html>'
>
> DEBUG The CA status is: check interrupted due to error: Retrieving CA
> status failed with status 500
>
> DEBUG Waiting for CA to start...
>
> ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
> run command ipa-server-upgrade manually.
>
> File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line
> 172, in execute
>
> return_value = self.run()
>
> File
> "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade
> .py",
> line 48, in run
>
> raise admintool.ScriptError(str(e))
>
> The ipa-server-upgrade command failed, exception: ScriptError: CA did
> not start in 300.0s
>
> ERROR CA did not start in 300.0s
>
> --
>
> With command "ipactl start --ignore-service-failures" I can start all
> the services but pki-tomcatd.
>
> --
>
> Directory Service: RUNNING
>
> krb5kdc Service: RUNNING
>
> kadmin Service: RUNNING
>
> named Service: RUNNING
>
> httpd Service: RUNNING
>
> pki-tomcatd Service: STOPPED
>
> smb Service: RUNNING
>
> winbind Service: RUNNING
>
> ipa-otpd Service: RUNNING
>
> ipa-dnskeysyncd Service: RUNNING
>
> ipa: INFO: The ipactl command was successful
>
> --
>
> Suggested resolution to above problem doesn't help me since the LDAP
> and NSS DB seem to have same certificates (some difference in wrapping
> but the string is same if I take out the line breaks) and even the
> serial number matches.
>
> --
>
> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
> cert-pki-ca' -a
>
> -----BEGIN CERTIFICATE-----
>
> MIIDjD.
>
> .Prh2G
>
> -----END CERTIFICATE-----
>
> certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
> |grep Serial
>
> Serial Number: 4 (0x4)
>
> ldapsearch -LLL -D 'cn=directory manager' -W -b
> uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
>
> Enter LDAP Password:
>
> dn: uid=pkidbuser,ou=people,o=ipaca
>
> userCertificate:: MIIDjD.
>
> .Prh2
>
> G
>
> description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
>
> O=<<REALM>>
>
> seeAlso: CN=CA Subsystem,O=<<REALM>>
>
> --
>
> And here's where my actual knowledge of things end. I've been trying
> to figure out all kind of logs (tomcat, Kerberos, directory server, .)
> but haven't found a solid reason for it. I'm starting to believe this
> is a certificate issue, because although "getcert list" tells me that
> the certificate status is "Monitoring" on all certificates the expiry
> date is already in the past (current date 20.6.2018, certificate
> expiry
> 21.03.2018) on 4 certificates and it won't update even if I resubmit
> it or delete certificate and manually redo it (it got the same date as
> the "old ones"). The "main certs" ("caSigningCert
cert-pki-ca",
> "Server-Cert cert-pki-ca" and two directory server certs) are valid
> for years (until
> 2020+).
>
> --
>
> Request ID '20160331084234':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC
> ert cert-pki-ca',token='NSS Certificate DB',pin set
>
> certificate:
> type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC
> ert cert-pki-ca',token='NSS Certificate DB'
>
> CA: dogtag-ipa-ca-renew-agent
>
> issuer: CN=Certificate Authority,O=<<REALM>>
>
> subject: CN=OCSP Subsystem,O=<<REALM>>
>
> expires: 2018-03-21 09:42:04 UTC
>
> key usage:
> digitalSignature,nonRepudiation,keyCertSign,cRLSign
>
> eku: id-kp-OCSPSigning
>
> pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
>
> post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
> "ocspSigningCert cert-pki-ca"
>
> track: yes
>
> auto-renew: yes
>
> Request ID '20160331085008':
>
> status: MONITORING
>
> stuck: no
>
> key pair storage:
>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N
> SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
>
> certificate:
>
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N
> SS
> Certificate DB'
>
> CA: IPA
>
> issuer: CN=Certificate Authority,O=<<REALM>>
>
> subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
>
> expires: 2020-03-04 09:58:23 UTC
>
> principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
>
> key usage:
> digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
>
> eku: id-kp-serverAuth,id-kp-clientAuth
>
> pre-save command:
>
> post-save command: /usr/lib64/ipa/certmonger/restart_httpd
>
> track: yes
>
> auto-renew: yes
>
> --
>
> Has anyone else bumped into same kind of issues? Any ideas where I
> should continue looking? I'm starting to run out of ideas.
>
> Eemeli Jokinen
>
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
> To unsubscribe send an email to
> freeipa-users-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> List Guidelines:
>
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo
>
rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
>
Hi,
does your topology include multiple CA instances? You need first to find which master is
the CA renewal master:
ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent with the content
of the LDAP entries. If it is not the case, you need to repair the CA renewal master
first.
When the CA renewal master is OK, check if the replication is working with the other CA
instances, and repair the other masters.
HTH,
Flo
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedorah...