Hi!
We do have 2 IPA nodes configured but the second node has been down for some time. Tried
to update it to same version as node1:
- Won't start tells me to use ipa-server-upgrade
- Ipa-server-upgrade fails at start, doesn't start directory server
- /var/log/dirsrv/slapd-<<REALM>>/errors log has some acl_parse errors but
last row tells me
--
EMERG - main - Fatal Error---No ports specified. Exiting now.
--
On node1 ipa config-show results only
--
ipa: ERROR: did not receive Kerberos credentials
--
Eemeli
-----Original Message-----
From: Florence Blanc-Renaud [mailto:flo@redhat.com]
Sent: keskiviikko 20. kesäkuuta 2018 19.49
To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
Cc: Jokinen Eemeli <Eemeli.Jokinen(a)cinia.fi>
Subject: Re: [Freeipa-users] Problems after IPA upgrade: ipa-server-upgrade doesn't
complete, pki-tomcatd won't start
On 06/20/2018 01:53 PM, Jokinen Eemeli via FreeIPA-users wrote:
Hello all!
I have very similiar problem as this one:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedor
ahosted.org/thread/YU6TZHOJAV5QHHHPQWJHYX3FP4OHA37X/
ipa-server-upgrade fails as below
--
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that CA proxy configuration is correct]
IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run
command ipa-server-upgrade manually.
CA did not start in 300.0s
The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for
more information
--
And the log tells me that CA returns status 500
--
DEBUG Waiting for CA to start...
DEBUG request POST http://<<ipa1.fqdn>>:8080/ca/admin/ca/getStatus
<http://%3c%3cipa1.fqdn%3e%3e:8080/ca/admin/ca/getStatus>
DEBUG request body ''
DEBUG response status 500
DEBUG response headers Server: Apache-Coyote/1.1
Content-Type: text/html;charset=utf-8
Content-Language: en
Content-Length: 2208
Date: Fri, 15 Jun 2018 10:05:29 GMT
Connection: close
DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 -
Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
D76;font-size:22px;}
H2
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
D76;font-size:16px;}
H3
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
D76;font-size:14px;}
BODY
{font-family:Tahoma,Arial,sans-serif;color:black;background-color:whit
e;} B
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525
D76;}
P
{font-family:Tahoma,Arial,sans-serif;background:white;color:black;font
-size:12px;}A {color : black;}A.name {color : black;}HR {color :
#525D76;}--></style> </head><body><h1>HTTP Status 500 -
Subsystem
unavailable</h1><HR size="1"
noshade="noshade"><p><b>type</b>
Exception report</p><p><b>message</b> <u>Subsystem
unavailable</u></p><p><b>description</b> <u>The
server encountered an
internal error that prevented it from fulfilling this
request.</u></p><p><b>exception</b>
<pre>javax.ws.rs.ServiceUnavailableException: Subsystem
unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstrai
nts(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.Authenti
catorBase.invoke(AuthenticatorBase.java:500)\n\torg.apache.catalina.va
lves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.
catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.
apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445
)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(Abstract
Http11Processor.java:1087)\n\torg.apache.coyote.AbstractProtocol$Abstr
actConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.
tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\
n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto
r.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(Thr
eadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread
$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thre
ad.java:748)\n</pre></p><p><b>note</b>
<u>The full stack trace of the root cause is available in the Apache
Tomcat/7.0.76 logs.</u></p><HR size="1"
noshade="noshade"><h3>Apache
Tomcat/7.0.76</h3></body></html>'
DEBUG The CA status is: check interrupted due to error: Retrieving CA
status failed with status 500
DEBUG Waiting for CA to start...
ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and
run command ipa-server-upgrade manually.
File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line
172, in execute
return_value = self.run()
File
"/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade
.py",
line 48, in run
raise admintool.ScriptError(str(e))
The ipa-server-upgrade command failed, exception: ScriptError: CA did
not start in 300.0s
ERROR CA did not start in 300.0s
--
With command "ipactl start --ignore-service-failures" I can start all
the services but pki-tomcatd.
--
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
named Service: RUNNING
httpd Service: RUNNING
pki-tomcatd Service: STOPPED
smb Service: RUNNING
winbind Service: RUNNING
ipa-otpd Service: RUNNING
ipa-dnskeysyncd Service: RUNNING
ipa: INFO: The ipactl command was successful
--
Suggested resolution to above problem doesn't help me since the LDAP
and NSS DB seem to have same certificates (some difference in wrapping
but the string is same if I take out the line breaks) and even the
serial number matches.
--
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDjD.
.Prh2G
-----END CERTIFICATE-----
certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert cert-pki-ca'
|grep Serial
Serial Number: 4 (0x4)
ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate description seeAlso
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate:: MIIDjD.
.Prh2
G
description: 2;4;CN=Certificate Authority,O=<<REALM>>;CN=CA Subsystem,
O=<<REALM>>
seeAlso: CN=CA Subsystem,O=<<REALM>>
--
And here's where my actual knowledge of things end. I've been trying
to figure out all kind of logs (tomcat, Kerberos, directory server, .)
but haven't found a solid reason for it. I'm starting to believe this
is a certificate issue, because although "getcert list" tells me that
the certificate status is "Monitoring" on all certificates the expiry
date is already in the past (current date 20.6.2018, certificate
expiry
21.03.2018) on 4 certificates and it won't update even if I resubmit
it or delete certificate and manually redo it (it got the same date as
the "old ones"). The "main certs" ("caSigningCert
cert-pki-ca",
"Server-Cert cert-pki-ca" and two directory server certs) are valid
for years (until
2020+).
--
Request ID '20160331084234':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC
ert cert-pki-ca',token='NSS Certificate DB',pin set
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningC
ert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=OCSP Subsystem,O=<<REALM>>
expires: 2018-03-21 09:42:04 UTC
key usage:
digitalSignature,nonRepudiation,keyCertSign,cRLSign
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert
"ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160331085008':
status: MONITORING
stuck: no
key pair storage:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N
SS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='N
SS
Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=<<REALM>>
subject: CN=<<ipasrv1.fqdn>>,O=<<REALM>>
expires: 2020-03-04 09:58:23 UTC
principal name: HTTP/<<ipasrv1.fqdn>>@<<REALM>>
key usage:
digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/lib64/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
--
Has anyone else bumped into same kind of issues? Any ideas where I
should continue looking? I'm starting to run out of ideas.
Eemeli Jokinen
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/freeipa-users@lists.fedo
rahosted.org/message/5XER2RAII4UH5URIMPL3GFHVBD7B6YSM/
Hi,
does your topology include multiple CA instances? You need first to find which master is
the CA renewal master:
ipa config-show | grep "renewal master"
On this host, check that the certificates are still valid and consistent with the content
of the LDAP entries. If it is not the case, you need to repair the CA renewal master
first.
When the CA renewal master is OK, check if the replication is working with the other CA
instances, and repair the other masters.
HTH,
Flo