On 04/04/2018 12:37 PM, lejeczek via FreeIPA-users wrote:
On 04/04/18 09:36, Florence Blanc-Renaud wrote:
> On 04/03/2018 08:37 PM, lejeczek wrote:
>>
>>
>> On 29/03/18 12:43, Florence Blanc-Renaud wrote:
>>> On 03/28/2018 12:42 PM, lejeczek via FreeIPA-users wrote:
>>>> hi guys,
>>>>
>>>> I fail to troubleshoot this here:
>>>>
>>>> $ ipactl start --ignore-service-failures
>>>> Starting Directory Service
>>>> Starting krb5kdc Service
>>>> Starting kadmin Service
>>>> Starting named Service
>>>> Starting httpd Service
>>>> Starting ipa-custodia Service
>>>> Starting ntpd Service
>>>> Starting pki-tomcatd Service
>>>> Failed to start pki-tomcatd Service
>>>> Forced start, ignoring pki-tomcatd Service, continuing normal
>>>> operation
>>>
>>> Hi,
>>>
>>> pki-tomcatd may fail to start when the subsystemCert cert-pki-ca did
>>> not properly get renewed. Please find more information in this blog:
>>>
https://floblanc.wordpress.com/2017/09/11/troubleshooting-freeipa-pki-tom...
>>>
>>>
>>> Flo
>>>
>>>> Starting ipa-otpd Service
>>>> Starting ipa-dnskeysyncd Service
>>>> ipa: INFO: The ipactl command was successful
>>>>
>>>> logs in /var/log/pki/pki-tomcat:
>>>> localhost.2018-03-28.log
>>>> ...
>>>> Mar 28, 2018 11:35:14 AM org.apache.catalina.core.StandardHostValve
>>>> invoke
>>>> SEVERE: Exception Processing /ca/admin/ca/getStatus
>>>> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
>>>> ?????????????? at
>>>>
com.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)
>>>>
>>>> ?????????????? at
>>>>
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:500)
>>>>
>>>> ?????????????? at
>>>>
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
>>>>
>>>> ?????????????? at
>>>>
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
>>>>
>>>> ?????????????? at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
>>>>
>>>> ?????????????? at
>>>>
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
>>>>
>>>> ?????????????? at
>>>>
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
>>>>
>>>> ?????????????? at
>>>>
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
>>>>
>>>> ?????????????? at
>>>>
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
>>>>
>>>> ?????????????? at
>>>>
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
>>>>
>>>> ?????????????? at
>>>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>>>
>>>> ?????????????? at
>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>>>
>>>> ?????????????? at
>>>>
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>>>>
>>>> ?????????????? at java.lang.Thread.run(Thread.java:748)
>>>>
>>>>
>>>> in catalina.2018-03-28.log:
>>>> ...
>>>> Mar 28, 2018 11:41:35 AM org.apache.catalina.core.ContainerBase
>>>> backgroundProcess
>>>> WARNING: Exception processing realm
>>>> com.netscape.cms.tomcat.ProxyRealm@1e572093 background process
>>>> javax.ws.rs.ServiceUnavailableException: Subsystem unavailable
>>>> ?????? at
>>>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:137)
>>>>
>>>> ?????? at
>>>>
org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1356)
>>>>
>>>> ?????? at
>>>>
org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5958)
>>>>
>>>> ?????? at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1542)
>>>>
>>>> ?????? at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>> ?????? at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1552)
>>>>
>>>> ?????? at
>>>>
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1520)
>>>>
>>>> ?????? at java.lang.Thread.run(Thread.java:748)
>>>>
>>>>
>>>> Would you able to conclude anything from those errors? What might
>>>> be a problem?
>>>>
>>>> many thanks, L.
>>>> _______________________________________________
>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>> To unsubscribe send an email to
>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>
>>
>> I have followed those instructions from the link and it seems that
>> both certutil & ldap have the same certificate.
>> However I also see:
>>
>> $ sudo journalctl -lf -o cat -u dirsrv@
>> ...
>> GSSAPI client step 1
>> GSSAPI client step 1
>> GSSAPI client step 2
>> [03/Apr/2018:19:30:53.962565693 +0100] - ERR - slapi_ldap_bind -
>> Error: could not send startTLS request: error -11 (Connect error)
>> [03/Apr/2018:19:30:53.965606137 +0100] - ERR - slapi_ldap_bind -
>> Error: could not send startTLS request: error -11 (Connect error)
>> GSSAPI client step 1
>> GSSAPI client step 1
>> GSSAPI client step 1
>> ..
>>
>> and in /var/log/pki/pki-tomcat/ca/debug
>>
>> [03/Apr/2018:19:09:45][localhost-startStop-1]:
>> SSLClientCertificatSelectionCB: Entering!
>> [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate cert:
>> Server-Cert cert-pki-ca
>> [03/Apr/2018:19:09:45][localhost-startStop-1]: Candidate cert:
>> caSigningCert cert-pki-ca
>> [03/Apr/2018:19:09:45][localhost-startStop-1]:
>> SSLClientCertificateSelectionCB: returning: null
>
> Hi,
> it looks like the subsystemCert is not picked to authenticate to the
> LDAP server. Can you check if the content of
> /etc/pki/pki-tomcat/ca/CS.cfg is also consistent: this file contains
> an entry for ca.subsystem.cert=MII.. that should match the cert
> 'subsystemCert cert-pki-ca' stored in /etc/pki/pki-tomcat/alias/ and
> in LDAP.
>
> Flo
>
>> [03/Apr/2018:19:09:45][localhost-startStop-1]: SSL handshake happened
>> Could not connect to LDAP server host rider.private port 636 Error
>> netscape.ldap.LDAPException: Authentication failed (48)
>> at
>>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)
>>
>> at
>>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)
>>
>> at
>>
com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)
>>
>> at
>> com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:654)
>> at
>> com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1176)
>> at
>> com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:1082)
>> at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:572)
>> at com.netscape.certsrv.apps.CMS.init(CMS.java:189)
>> at com.netscape.certsrv.apps.CMS.start(CMS.java:1631)
>> at
>> com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)
>>
>> at javax.servlet.GenericServlet.init(GenericServlet.java:158)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>> at
>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)
>> at
>> org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
>> at
>> org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)
>> at
>> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)
>>
>> at
>> org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)
>>
>> at
>> org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1257)
>>
>> at
>> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1182)
>>
>> at
>> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1072)
>> at
>> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5368)
>>
>> at
>> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5660)
>>
>> at
>> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
>> at
>> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
>>
>> at
>> org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)
>>
>> at
>>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)
>>
>> at
>>
org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)
>>
>> at java.security.AccessController.doPrivileged(Native Method)
>> at
>> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)
>> at
>> org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
>> at
>> org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
>>
>> at
>> org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
>>
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ..
>
could this have some causation in expired certs?
Hi,
CA_WORKING means that certmonger's helper is trying to download the
certificate from LDAP, but does not find new certs.
In topologies with multiple servers, only one server is the renewal
master. When one of auditSigningCert cert-pki-ca, ocspSigningCert
cert-pki-ca, subsystemCert cert-pki-ca or caSigningCert cert-pki-ca
expires, the renewal master is the one that actually handles the
renewal, and the other masters simply download the new certs from LDAP.
You need to check which server is your renewal master (ipa config-show |
grep 'IPA CA renewal master'), then make sure that the certs were
properly renewed on this master (check consistency between
/etc/pki/pki-tomcat/alias, the certs in
cn=certificates,cn=ipa,cn=etc,$BASEDN, and the content in
/etc/pki/pki-tomcat/ca/CS.cfg).
Then check that replication is working between the renewal master and
the other masters. If the replication is broken, the certs will not be
copied on the other masters and the download will not detect new
certificates.
HTH,
Flo
$ getcert list | grep -E "Request
ID|status|certificate|expires"
Number of certificates and requests being tracked: 9.
Request ID '20170920090053':
status: MONITORING
certificate: type=FILE,location='/var/kerberos/krb5kdc/kdc.crt'
expires: 2018-09-20 09:00:53 UTC
certificate template/profile: KDCs_PKINIT_Certs
Request ID '20171221120303':
status: CA_WORKING
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert
cert-pki-ca',token='NSS Certificate DB'
expires: 2018-03-27 14:07:51 UTC
Request ID '20171221120304':
status: CA_WORKING
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert
cert-pki-ca',token='NSS
Certificate DB'
expires: 2018-03-27 14:07:50 UTC
Request ID '20171221120305':
status: CA_WORKING
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert
cert-pki-ca',token='NSS Certificate DB'
expires: 2018-03-27 14:07:51 UTC
Request ID '20171221120306':
status: MONITORING
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert
cert-pki-ca',token='NSS Certificate DB'
expires: 2036-04-06 14:07:49 UTC
Request ID '20171221120307':
status: CA_WORKING
certificate: type=FILE,location='/var/lib/ipa/ra-agent.pem'
expires: 2018-03-27 14:08:18 UTC
Request ID '20171221120308':
status: MONITORING
certificate:
type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert
cert-pki-ca',token='NSS Certificate DB'
expires: 2019-07-25 16:13:23 UTC
Request ID '20171221120309':
status: MONITORING
certificate:
type=NSSDB,location='/etc/dirsrv/slapd-PRIVATE-CCNR-CEB-PRIVATE-CAM-AC-UK',nickname='Server-Cert',token='NSS
Certificate DB'
expires: 2019-04-09 12:11:52 UTC
Request ID '20171221120310':
status: MONITORING
certificate:
type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS
Certificate DB'
expires: 2019-04-09 12:11:54 UTC
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org