Hi,

On Sat, Sep 3, 2022 at 6:40 AM Polavarapu Manideep Sai <manideep.sai@onmobile.com> wrote:

Hi Florence,

 

Done the same and tried installation for multiple times but same issue

 

Please find below response inline

 

Can you clean up the replica you're trying to install and start over, then send the most recent logs?  Done

- on the failing replica: ipa-server-install --uninstall -U    Done

- on the master: kinit admin; ipa server-del <replica> --force   Done

- on the failing replica: perform the installation with your usual method (either in a 2-step process with ipa-client-install/ipa-replica-install or in a single step with ipa-replica-install).  Done with below command

“ipa-replica-install -n ipa.subdomain.com --hostname=dirpav01.ipa.subdomain.com  --server=aaa01.ipa.subdomain.com --realm=IPA.SUBDOMAIN.COM -P admin -w XXXXXXX --no-host-dns --setup-ca --setup-dns --mkhomedir --auto-reverse --no-forwarders”

 

-Also provide the timezone of the replica so that we can translate all the timestamps in UTC time.

 

4. Time Zone

 

[root@dirpav01 ~]# timedatectl

      Local time: Fri 2022-09-02 20:11:53 CEST

  Universal time: Fri 2022-09-02 18:11:53 UTC

        RTC time: Fri 2022-09-02 18:11:52

       Time zone: Europe/Madrid (CEST, +0200)

     NTP enabled: no

NTP synchronized: yes

RTC in local TZ: no

      DST active: yes

Last DST change: DST began at

                  Sun 2022-03-27 01:59:59 CET

                  Sun 2022-03-27 03:00:00 CEST

Next DST change: DST ends (the clock jumps one hour backwards) at

                  Sun 2022-10-30 02:59:59 CEST

                  Sun 2022-10-30 02:00:00 CET

[root@dirpav01 ~]#

 

 

 

=======================

Replica Installation:

=======================

 

 

[root@dirpav01 ~]# ipa-replica-install -n ipa.subdomain.com --hostname=dirpav01.ipa.subdomain.com  --server=aaa01.ipa.subdomain.com --realm=IPA.SUBDOMAIN.COM -P admin -w Adm@onm0# --no-host-dns --setup-ca --setup-dns --mkhomedir --auto-reverse --no-forwarders

Configuring client side components

Client hostname: dirpav01.ipa.subdomain.com

Realm: IPA.SUBDOMAIN.COM

DNS Domain: ipa.subdomain.com

IPA Server: aaa01.ipa.subdomain.com

BaseDN: dc=ipa,dc=subdomain,dc=com

 

Skipping synchronizing time with NTP server.

Successfully retrieved CA cert

    Subject:     CN=Certificate Authority,O=IPA.SUBDOMAIN.COM

    Issuer:      CN=Certificate Authority,O=IPA.SUBDOMAIN.COM

    Valid From:  2018-04-12 14:15:30

    Valid Until: 2038-04-12 14:15:30

 

Enrolled in IPA realm IPA.SUBDOMAIN.COM

Created /etc/ipa/default.conf

New SSSD config will be created

Configured sudoers in /etc/nsswitch.conf

Configured /etc/sssd/sssd.conf

Configured /etc/krb5.conf for IPA realm IPA.SUBDOMAIN.COM

trying https://aaa01.ipa.subdomain.com/ipa/json

[try 1]: Forwarding 'schema' to json server 'https://aaa01.ipa.subdomain.com/ipa/json'

trying https://aaa01.ipa.subdomain.com/ipa/session/json

[try 1]: Forwarding 'ping' to json server 'https://aaa01.ipa.subdomain.com/ipa/session/json'

[try 1]: Forwarding 'ca_is_enabled' to json server 'https://aaa01.ipa.subdomain.com/ipa/session/json'

Systemwide CA database updated.

DNS query for dirpav01.ipa.subdomain.com. A failed: The DNS operation timed out after 30.0018370152 seconds

DNS resolution for hostname dirpav01.ipa.subdomain.com failed: The DNS operation timed out after 30.0018370152 seconds

Failed to update DNS records.


The server aaa01 and the replica dirpav01 need to be resolvable from both machines and have their reverse records properly set.

HTH,
flo

Missing A/AAAA record(s) for host dirpav01.ipa.subdomain.com: 10.26.60.179.

Missing reverse record(s) for address(es): 10.26.60.179.

Adding SSH public key from /etc/ssh/ssh_host_rsa_key.pub

Adding SSH public key from /etc/ssh/ssh_host_ecdsa_key.pub

Adding SSH public key from /etc/ssh/ssh_host_ed25519_key.pub

[try 1]: Forwarding 'host_mod' to json server 'https://aaa01.ipa.subdomain.com/ipa/session/json'

Could not update DNS SSHFP records.

SSSD enabled

Configured /etc/openldap/ldap.conf

Configured /etc/ssh/ssh_config

Configured /etc/ssh/sshd_config

Configuring ipa.subdomain.com as NIS domain.

Client configuration complete.

The ipa-client-install command was successful

 

Warning: skipping DNS resolution of host dirpav01.ipa.subdomain.com

Warning: skipping DNS resolution of host aaa01.ipa.subdomain.com

Run connection check to master

Connection check OK

Configuring NTP daemon (ntpd)

  [1/4]: stopping ntpd

  [2/4]: writing configuration

  [3/4]: configuring ntpd to start on boot

  [4/4]: starting ntpd

Done configuring NTP daemon (ntpd).

Configuring directory server (dirsrv). Estimated time: 30 seconds

  [1/42]: creating directory server instance

  [2/42]: enabling ldapi

  [3/42]: configure autobind for root

  [4/42]: stopping directory server

  [5/42]: updating configuration in dse.ldif

  [6/42]: starting directory server

  [7/42]: adding default schema

  [8/42]: enabling memberof plugin

  [9/42]: enabling winsync plugin

  [10/42]: configure password logging

  [11/42]: configuring replication version plugin

  [12/42]: enabling IPA enrollment plugin

  [13/42]: configuring uniqueness plugin

  [14/42]: configuring uuid plugin

  [15/42]: configuring modrdn plugin

  [16/42]: configuring DNS plugin

  [17/42]: enabling entryUSN plugin

  [18/42]: configuring lockout plugin

  [19/42]: configuring topology plugin

  [20/42]: creating indices

  [21/42]: enabling referential integrity plugin

  [22/42]: configuring certmap.conf

  [23/42]: configure new location for managed entries

  [24/42]: configure dirsrv ccache

  [25/42]: enabling SASL mapping fallback

  [26/42]: restarting directory server

  [27/42]: creating DS keytab

  [28/42]: ignore time skew for initial replication

  [29/42]: setting up initial replication

Starting replication, please wait until this has completed.

Update in progress, 31 seconds elapsed

Update succeeded

 

  [30/42]: prevent time skew after initial replication

  [31/42]: adding sasl mappings to the directory

  [32/42]: updating schema

  [33/42]: setting Auto Member configuration

  [34/42]: enabling S4U2Proxy delegation

  [35/42]: initializing group membership

  [36/42]: adding master entry

  [37/42]: initializing domain level

  [38/42]: configuring Posix uid/gid generation

  [39/42]: adding replication acis

  [40/42]: activating sidgen plugin

  [41/42]: activating extdom plugin

  [42/42]: configuring directory to start on boot

Done configuring directory server (dirsrv).

Configuring Kerberos KDC (krb5kdc)

  [1/5]: configuring KDC

  [2/5]: adding the password extension to the directory

  [3/5]: creating anonymous principal

  [4/5]: starting the KDC

  [5/5]: configuring KDC to start on boot

Done configuring Kerberos KDC (krb5kdc).

Configuring kadmin

  [1/2]: starting kadmin

  [2/2]: configuring kadmin to start on boot

Done configuring kadmin.

Configuring directory server (dirsrv)

  [1/3]: configuring TLS for DS instance

  [2/3]: importing CA certificates from LDAP

  [3/3]: restarting directory server

Done configuring directory server (dirsrv).

Configuring the web interface (httpd)

  [1/22]: stopping httpd

  [2/22]: setting mod_nss port to 443

  [3/22]: setting mod_nss cipher suite

  [4/22]: setting mod_nss protocol list to TLSv1.2

  [5/22]: setting mod_nss password file

  [6/22]: enabling mod_nss renegotiate

  [7/22]: disabling mod_nss OCSP

  [8/22]: adding URL rewriting rules

  [9/22]: configuring httpd

  [10/22]: setting up httpd keytab

  [11/22]: configuring Gssproxy

  [12/22]: setting up ssl

  [13/22]: configure certmonger for renewals

  [14/22]: importing CA certificates from LDAP

  [15/22]: publish CA cert

  [16/22]: clean up any existing httpd ccaches

  [17/22]: configuring SELinux for httpd

  [18/22]: create KDC proxy config

  [19/22]: enable KDC proxy

  [20/22]: starting httpd

  [21/22]: configuring httpd to start on boot

  [22/22]: enabling oddjobd

Done configuring the web interface (httpd).

Configuring ipa-otpd

  [1/2]: starting ipa-otpd

  [2/2]: configuring ipa-otpd to start on boot

Done configuring ipa-otpd.

Configuring ipa-custodia

  [1/4]: Generating ipa-custodia config file

  [2/4]: Generating ipa-custodia keys

  [3/4]: starting ipa-custodia

  [4/4]: configuring ipa-custodia to start on boot

Done configuring ipa-custodia.

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes

  [1/30]: creating certificate server db

  [2/30]: setting up initial replication

Starting replication, please wait until this has completed.

Update in progress, 30 seconds elapsed

Update succeeded

 

  [3/30]: creating ACIs for admin

  [4/30]: creating installation admin user

  [5/30]: configuring certificate server instance

  [6/30]: secure AJP connector

  [7/30]: reindex attributes

  [8/30]: exporting Dogtag certificate store pin

  [9/30]: stopping certificate server instance to update CS.cfg

  [10/30]: backing up CS.cfg

  [11/30]: disabling nonces

  [12/30]: set up CRL publishing

  [13/30]: enable PKIX certificate path discovery and validation

  [14/30]: destroying installation admin user

  [15/30]: starting certificate server instance

  [16/30]: Finalize replication settings

  [17/30]: configure certmonger for renewals

  [18/30]: Importing RA key

  [19/30]: setting audit signing renewal to 2 years

  [20/30]: restarting certificate server

  [21/30]: authorizing RA to modify profiles

  [22/30]: authorizing RA to manage lightweight CAs

  [23/30]: Ensure lightweight CAs container exists

  [24/30]: configure certificate renewals

  [25/30]: configure Server-Cert certificate renewal

  [26/30]: Configure HTTP to proxy connections

  [27/30]: restarting certificate server

  [28/30]: updating IPA configuration

  [29/30]: enabling CA instance

  [30/30]: configuring certmonger renewal for lightweight CAs

Done configuring certificate server (pki-tomcatd).

Your system may be partly configured.

Run /usr/sbin/ipa-server-install --uninstall to clean up.

 

ipapython.admintool: ERROR    CA did not start in 300.0s

ipapython.admintool: ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

[root@dirpav01 ~]#

 

 

 

================================

/var/log/pki/pki-tomcat/ca/debug

================================

 

 

[02/Sep/2022:20:41:02][localhost-startStop-1]: SSLClientCertificateSelectionCB: Setting desired cert nickname to: subsystemCert cert-pki-ca

[02/Sep/2022:20:41:02][localhost-startStop-1]: ldapconn/PKISocketFactory.makeSSLSocket:  set client auth cert nickname subsystemCert cert-pki-ca

[02/Sep/2022:20:41:02][localhost-startStop-1]: SSLClientCertificatSelectionCB: Entering!

[02/Sep/2022:20:41:02][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca

[02/Sep/2022:20:41:02][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca

[02/Sep/2022:20:41:02][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca

[02/Sep/2022:20:41:02][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca

[02/Sep/2022:20:41:02][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: begins

[02/Sep/2022:20:41:02][localhost-startStop-1]: SignedAuditLogger: event CLIENT_ACCESS_SESSION_ESTABLISH

[02/Sep/2022:20:41:02][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: CS_CLIENT_ACCESS_SESSION_ESTABLISH_SUCCESS

[02/Sep/2022:20:41:02][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: clientIP=10.26.60.179 serverIP=10.26.60.179 serverPort=31746

[02/Sep/2022:20:41:02][localhost-startStop-1]: SSL handshake happened

Could not connect to LDAP server host dirpav01.ipa.subdomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)

        at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)

        at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)

        at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)

        at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667)

        at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1054)

        at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:960)

        at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:566)

        at com.netscape.certsrv.apps.CMS.init(CMS.java:194)

        at com.netscape.certsrv.apps.CMS.start(CMS.java:1461)

        at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)

        at javax.servlet.GenericServlet.init(GenericServlet.java:158)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)

        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)

        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)

        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)

        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)

        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218)

        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)

        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)

        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377)

        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)

        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)

        at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)

        at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)

        at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)

        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)

        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)

        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:750)

Internal Database Error encountered: Could not connect to LDAP server host dirpav01.ipa.subdomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)

        at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)

        at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1054)

        at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:960)

        at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:566)

        at com.netscape.certsrv.apps.CMS.init(CMS.java:194)

       at com.netscape.certsrv.apps.CMS.start(CMS.java:1461)

        at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)

        at javax.servlet.GenericServlet.init(GenericServlet.java:158)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)

        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)

        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:320)

        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:175)

        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:124)

        at org.apache.catalina.core.StandardWrapper.initServlet(StandardWrapper.java:1218)

        at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1174)

        at org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:1066)

        at org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:5377)

        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)

        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)

        at org.apache.catalina.core.ContainerBase.access$000(ContainerBase.java:133)

        at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:156)

        at org.apache.catalina.core.ContainerBase$PrivilegedAddChild.run(ContainerBase.java:145)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:873)

        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)

        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)

        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:750)

[02/Sep/2022:20:41:02][localhost-startStop-1]: CMS.start(): shutdown server

[02/Sep/2022:20:41:02][localhost-startStop-1]: CMSEngine.shutdown()

[root@dirpav01 ~]#

 

 

================================

/var/log/ipareplica-install.log

================================

 

 

2022-09-02T18:42:31Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'

2022-09-02T18:42:31Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

2022-09-02T18:42:31Z DEBUG Waiting for CA to start...

2022-09-02T18:42:32Z DEBUG request POST http://dirpav01.ipa.subdomain.com:8080/ca/admin/ca/getStatus

2022-09-02T18:42:32Z DEBUG request body ''

2022-09-02T18:42:32Z DEBUG response status 500

2022-09-02T18:42:32Z DEBUG response headers Server: Apache-Coyote/1.1

Content-Type: text/html;charset=utf-8

Content-Language: en

Content-Length: 2208

Date: Fri, 02 Sep 2022 18:42:32 GMT

Connection: close

 

2022-09-02T18:42:32Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'

2022-09-02T18:42:32Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

2022-09-02T18:42:32Z DEBUG Waiting for CA to start...

2022-09-02T18:42:33Z DEBUG request POST http://dirpav01.ipa.subdomain.com:8080/ca/admin/ca/getStatus

2022-09-02T18:42:33Z DEBUG request body ''

2022-09-02T18:42:34Z DEBUG response status 500

2022-09-02T18:42:34Z DEBUG response headers Server: Apache-Coyote/1.1

Content-Type: text/html;charset=utf-8

Content-Language: en

Content-Length: 2208

Date: Fri, 02 Sep 2022 18:42:34 GMT

Connection: close

 

2022-09-02T18:42:34Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'

2022-09-02T18:42:34Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

2022-09-02T18:42:34Z DEBUG Waiting for CA to start...

2022-09-02T18:42:35Z DEBUG request POST http://dirpav01.ipa.subdomain.com:8080/ca/admin/ca/getStatus

2022-09-02T18:42:35Z DEBUG request body ''

2022-09-02T18:42:35Z DEBUG response status 500

2022-09-02T18:42:35Z DEBUG response headers Server: Apache-Coyote/1.1

Content-Type: text/html;charset=utf-8

Content-Language: en

Content-Length: 2208

Date: Fri, 02 Sep 2022 18:42:35 GMT

Connection: close

 

2022-09-02T18:42:35Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'

2022-09-02T18:42:35Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

2022-09-02T18:42:35Z DEBUG Waiting for CA to start...

2022-09-02T18:42:36Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute

    return_value = self.run()

  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run

    return cfgr.run()

  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 360, in run

    return self.execute()

  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 386, in execute

    for rval in self._executor():

  File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner

  File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py", line 464, in start

    self.service.start(instance_name, capture_output=capture_output, wait=wait)

  File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 192, in start

    self.wait_until_running()

  File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 186, in wait_until_running

    raise RuntimeError('CA did not start in %ss' % timeout)

 

2022-09-02T18:42:36Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: CA did not start in 300.0s

2022-09-02T18:42:36Z ERROR CA did not start in 300.0s

2022-09-02T18:42:36Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

[root@dirpav01 ~]#

 

 

Sai

 

From: Florence Blanc-Renaud <flo@redhat.com>
Sent: Friday, September 2, 2022 5:37 PM
To: Polavarapu Manideep Sai <manideep.sai@onmobile.com>
Cc: Rob Crittenden <rcritten@redhat.com>; FreeIPA users list <freeipa-users@lists.fedorahosted.org>
Subject: Re: [Freeipa-users] Re: Free IPA Replica server retrieving two certificates from the IPA master server while installing IPA replica and installation fails

 

CAUTION. This email originated from outside the organization. Please exercise caution before clicking on links or attachments in case of suspicion or unknown senders.

Hi,

 

On Thu, Sep 1, 2022 at 7:47 PM Polavarapu Manideep Sai <manideep.sai@onmobile.com> wrote:

Hi Florence/Rob

 

Upon your advice, I have removed the certificate from the IPA master, Now IPA Replica retrieving one certificate from the IPA master as shown below

 

Facing another IPA Replica installation issue after deleting/removing the certificate from the IPA master server, please help us on this, please let us know anymore information required on this

 

PFB Replica installation Logs

 

==============================

/var/log/ipaclient-install.log :

==============================

 

2022-09-01T17:03:00Z DEBUG stderr=

2022-09-01T17:03:00Z DEBUG trying to retrieve CA cert via LDAP from aaa01.ipa.subdomain.com

2022-09-01T17:03:01Z DEBUG retrieving schema for SchemaCache url=ldap://aaa01.ipa.subdomain.com:389 conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f840831d3f8>

2022-09-01T17:03:02Z INFO Successfully retrieved CA cert

    Subject:     CN=Certificate Authority,O=IPA.SUBDOMAIN.COM

    Issuer:      CN=Certificate Authority,O=IPA.SUBDOMAIN.COM

    Valid From:  2018-04-12 14:15:30

    Valid Until: 2038-04-12 14:15:30

 

2022-09-01T17:03:02Z DEBUG Starting external process

2022-09-01T17:03:02Z DEBUG args=/usr/sbin/ipa-join -s aaa01.ipa.subdomain.com -b dc=ipa,dc=subdomain,dc=com -h dirpav01.ipa.subdomain.com -f

2022-09-01T17:03:07Z DEBUG Process finished, return code=0

2022-09-01T17:03:07Z DEBUG stdout=

2022-09-01T17:03:07Z DEBUG stderr=Keytab successfully retrieved and stored in: /etc/krb5.keytab

Certificate subject base is: O=IPA.SUBDOMAIN.COM

 

2022-09-01T17:03:07Z INFO Enrolled in IPA realm IPA.SUBDOMAIN.COM

2022-09-01T17:03:07Z DEBUG Starting external process

2022-09-01T17:03:07Z DEBUG args=/usr/bin/kdestroy

2022-09-01T17:03:07Z DEBUG Process finished, return code=0

2022-09-01T17:03:07Z DEBUG stdout=

2022-09-01T17:03:07Z DEBUG stderr=

 

 

======================================

Replica installation without debugging :

======================================

 

Done configuring ipa-custodia.

Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes

  [1/30]: creating certificate server db

  [2/30]: setting up initial replication

Starting replication, please wait until this has completed.

Update in progress, 30 seconds elapsed

Update succeeded

 

  [3/30]: creating ACIs for admin

  [4/30]: creating installation admin user

  [5/30]: configuring certificate server instance

  [6/30]: secure AJP connector

  [7/30]: reindex attributes

  [8/30]: exporting Dogtag certificate store pin

  [9/30]: stopping certificate server instance to update CS.cfg

  [10/30]: backing up CS.cfg

  [11/30]: disabling nonces

  [12/30]: set up CRL publishing

  [13/30]: enable PKIX certificate path discovery and validation

  [14/30]: destroying installation admin user

  [15/30]: starting certificate server instance

  [16/30]: Finalize replication settings

  [17/30]: configure certmonger for renewals

  [18/30]: Importing RA key

  [19/30]: setting audit signing renewal to 2 years

  [20/30]: restarting certificate server

  [21/30]: authorizing RA to modify profiles

  [22/30]: authorizing RA to manage lightweight CAs

  [23/30]: Ensure lightweight CAs container exists

  [24/30]: configure certificate renewals

  [25/30]: configure Server-Cert certificate renewal

  [26/30]: Configure HTTP to proxy connections

  [27/30]: restarting certificate server

  [28/30]: updating IPA configuration

  [29/30]: enabling CA instance

  [30/30]: configuring certmonger renewal for lightweight CAs

Done configuring certificate server (pki-tomcatd).

Your system may be partly configured.

Run /usr/sbin/ipa-server-install --uninstall to clean up.

 

ipapython.admintool: ERROR    CA did not start in 300.0s

ipapython.admintool: ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

 

 

 

 

================================

/var/log/ipareplica-install.log

================================

 

 

2022-09-01T14:35:58Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'

2022-09-01T14:35:58Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

2022-09-01T14:35:58Z DEBUG Waiting for CA to start...

2022-09-01T14:35:59Z DEBUG request POST http://dirpav01.ipa.subdomain.com:8080/ca/admin/ca/getStatus

2022-09-01T14:35:59Z DEBUG request body ''

2022-09-01T14:35:59Z DEBUG response status 500

2022-09-01T14:35:59Z DEBUG response headers Server: Apache-Coyote/1.1^M

Content-Type: text/html;charset=utf-8^M

Content-Language: en^M

Content-Length: 2208^M

Date: Thu, 01 Sep 2022 14:35:59 GMT^M

Connection: close^M

 

2022-09-01T14:35:59Z DEBUG response body '<html><head><title>Apache Tomcat/7.0.76 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - Subsystem unavailable</h1><HR size="1" noshade="noshade"><p><b>type</b> Exception report</p><p><b>message</b> <u>Subsystem unavailable</u></p><p><b>description</b> <u>The server encountered an internal error that prevented it from fulfilling this request.</u></p><p><b>exception</b> <pre>javax.ws.rs.ServiceUnavailableException: Subsystem unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:145)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)\n\torg.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)\n\torg.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)\n\torg.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)\n\torg.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:750)\n</pre></p><p><b>note</b> <u>The full stack trace of the root cause is available in the Apache Tomcat/7.0.76 logs.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.76</h3></body></html>'

2022-09-01T14:35:59Z DEBUG The CA status is: check interrupted due to error: Retrieving CA status failed with status 500

2022-09-01T14:35:59Z DEBUG Waiting for CA to start...

2022-09-01T14:36:00Z DEBUG   File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute

    return_value = self.run()

  File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run

   File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 186, in wait_until_running

    raise RuntimeError('CA did not start in %ss' % timeout)

 

2022-09-01T14:36:00Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: CA did not start in 300.0s

2022-09-01T14:36:00Z ERROR CA did not start in 300.0s

2022-09-01T14:36:00Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

 

 

 

The logs are a bit confusing, the client install logs have timestamps around 2022-09-01T17:03:07Z but replica-install around 2022-09-01T14:36:00Z which is earlier? Same comment for the tomcat logs around 01/Sep/2022:16:45:21 (pki logs use the local timezone while client and repl logs use UTC times, but the times are completely unrelated here).

 

Can you clean up the replica you're trying to install and start over, then send the most recent logs?

- on the failing replica: ipa-server-install --uninstall -U

- on the master: kinit admin; ipa server-del <replica> --force

- on the failing replica: perform the installation with your usual method (either in a 2-step process with ipa-client-install/ipa-replica-install or in a single step with ipa-replica-install).

Also provide the timezone of the replica so that we can translate all the timestamps in UTC time.

 

flo

 

=================================

/var/log/pki/pki-tomcat/ca/debug :

=================================

 

 

[01/Sep/2022:16:45:21][localhost-startStop-1]: Candidate cert: ocspSigningCert cert-pki-ca

[01/Sep/2022:16:45:21][localhost-startStop-1]: Candidate cert: subsystemCert cert-pki-ca

[01/Sep/2022:16:45:21][localhost-startStop-1]: SSLClientCertificateSelectionCB: desired cert found in list: subsystemCert cert-pki-ca

[01/Sep/2022:16:45:21][localhost-startStop-1]: SSLClientCertificateSelectionCB: returning: subsystemCert cert-pki-ca

[01/Sep/2022:16:45:21][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: begins

[01/Sep/2022:16:45:21][localhost-startStop-1]: SignedAuditLogger: event CLIENT_ACCESS_SESSION_ESTABLISH

[01/Sep/2022:16:45:21][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: CS_CLIENT_ACCESS_SESSION_ESTABLISH_SUCCESS

[01/Sep/2022:16:45:21][localhost-startStop-1]: PKIClientSocketListener.handshakeCompleted: clientIP=10.26.60.179 serverIP=10.26.60.179 serverPort=31746

[01/Sep/2022:16:45:21][localhost-startStop-1]: SSL handshake happened

Could not connect to LDAP server host dirpav01.ipa.subdomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)

        at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.makeConnection(LdapBoundConnFactory.java:205)

        at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:166)

        at com.netscape.cmscore.ldapconn.LdapBoundConnFactory.init(LdapBoundConnFactory.java:130)

        at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:667)

        at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1054)

        at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:960)

        at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:566)

        at com.netscape.certsrv.apps.CMS.init(CMS.java:194)

        at com.netscape.certsrv.apps.CMS.start(CMS.java:1461)

        at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)

        at javax.servlet.GenericServlet.init(GenericServlet.java:158)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:288)

        at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:285)

                                        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)

        at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)

        at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:750)

Internal Database Error encountered: Could not connect to LDAP server host dirpav01.ipa.subdomain.com port 636 Error netscape.ldap.LDAPException: Authentication failed (48)

        at com.netscape.cmscore.dbs.DBSubsystem.init(DBSubsystem.java:689)

        at com.netscape.cmscore.apps.CMSEngine.initSubsystem(CMSEngine.java:1054)

        at com.netscape.cmscore.apps.CMSEngine.initSubsystems(CMSEngine.java:960)

        at com.netscape.cmscore.apps.CMSEngine.init(CMSEngine.java:566)

        at com.netscape.certsrv.apps.CMS.init(CMS.java:194)

        at com.netscape.certsrv.apps.CMS.start(CMS.java:1461)

        at com.netscape.cms.servlet.base.CMSStartServlet.init(CMSStartServlet.java:117)

        at javax.servlet.GenericServlet.init(GenericServlet.java:158)

                                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:750)

[01/Sep/2022:16:45:21][localhost-startStop-1]: CMS.start(): shutdown server

[01/Sep/2022:16:45:21][localhost-startStop-1]: CMSEngine.shutdown()

 

 

Sai

 

 

From: Florence Blanc-Renaud via FreeIPA-users <freeipa-users@lists.fedorahosted.org>
Sent: Wednesday, August 31, 2022 12:28 PM
To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>
Cc: Rob Crittenden <rcritten@redhat.com>; Polavarapu Manideep Sai <manideep.sai@onmobile.com>; Florence Blanc-Renaud <flo@redhat.com>
Subject: [Freeipa-users] Re: Free IPA Replica server retrieving two certificates from the IPA master server while installing IPA replica and installation fails

 

CAUTION. This email originated from outside the organization. Please exercise caution before clicking on links or attachments in case of suspicion or unknown senders.

Hi,

I'm replying to the same questions posted on my blog:

Hi floblanc,

Thank you for the reply,

I have a few queries, can you please clarify

1. should we run ipa-cert-update on IPA master server also and then after on all IPA replica server and their clients ?

Yes, ipa-certupdate has to be run on all the machines enrolled into IPA.

2. Do we need to consider only one common name i.e. “cn=directory manager” as we have two one is LADP and other one is for HTTP

dbm:/etc/dirsrv/slapd-IPA-ONMOBILE-COM/
dbm:/etc/httpd/alias

ldapsearch -D “cn=directory manager” -W -b cn=certificates,cn=ipa,cn=etc,dc=ipa,dc=example,dc=com “(&(objectClass=ipaCertificate)(objectClass=pkiCA))”

Refer to ldapsearch man page to understand the options:

- the -D "cn=directory manager" option means that the LDAP operations will be authenticated with the user Directory Manager. When you installed the first IPA server with ipa-server-install, this user was created with the password provided with ipa-server-install -p|--ds-password DM_PASSWORD.

- the -W option means "prompt for password"

- the -b option specifies a search base. The CA certificates are stored below cn=certificates,cn=ipa,cn=etc,dc=ipa,dc=example,dc=com, the search needs to target this search base

- “(&(objectClass=ipaCertificate)(objectClass=pkiCA))” is the search filter allowing to find CA certificates

 

This single search allows to retrieve all the CA certificates, one ldap entry for each certificate.

Any other common name for HTTP:

ldapsearch -D “cn=?” -W -b cn=certificates,cn=ipa,cn=etc,dc=ipa,dc=example,dc=com “(&(objectClass=ipaCertificate)(objectClass=pkiCA))”

Or else this is the only query to search the ipaCertificate in whole ldap database?

if i want to search the all occurrence of this invalid certificate in the whole server/database, how can we achieve this

3. I have a infrastructure with one IPA master and 13 IPA Replicas, if i delete the certificate in IPA Master and run ipa-certupdate, and again run ipa-certupdate on 13 IPA Replica servers, and its clients, i hope there will not be any issue after changes and also pki-tomcatd.target service will be running

If the LDAP entry corresponding to the certificate is deleted on the IPA master, the replication will propagate this deletion to the other replicas. This means the entry will be removed from all the LDAP servers.

When ipa-certupdate is run, the list of CA certificates is refreshed (re-read from LDAP) and updated on the local NSS Databases.

 

HTH,

flo

Or do you suggest any other better way without any impact on services further as it is production setup

Note: As we deleted last time then pki-tomcat.target service was stopped and not started [we didn’t run ipa-certupdate on IPA Master]

How can we check all occurrence of this invalid certificate in IPA master server

 

On Tue, Aug 30, 2022 at 8:09 PM Polavarapu Manideep Sai via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:

Hi Rob,

Can you please help me on this

Regards
ManideepSai

-----Original Message-----
From: Rob Crittenden <rcritten@redhat.com>
Sent: Tuesday, August 30, 2022 11:36 PM
To: FreeIPA users list <freeipa-users@lists.fedorahosted.org>
Cc: Polavarapu Manideep Sai <manideep.sai@onmobile.com>
Subject: Re: [Freeipa-users] Free IPA Replica server retrieving two certificates from the IPA master server while installing IPA replica and installation fails


CAUTION. This email originated from outside the organization. Please exercise caution before clicking on links or attachments in case of suspicion or unknown senders.




Polavarapu Manideep Sai via FreeIPA-users wrote:
> Hi Team,
>
>
>
> Need help from freeipa,
>
>
>
> Free IPA Replica server retrieving two certificates from the IPA master
> server while installing IPA replica and installation fails
>
>
>
> please check the below issue and let us know the fix and please let us
> know if any more details required
>
>
>
> Master server: aaa01
>
> Replica server1:  dir01 (currently installing replica server )
>
> Replica server2: dirus02 (which was a replica server previously that has
> been removed from replication)
>
>
>
>
>
> As noticed while installing ipa replica server, replica server
> retrieving two certificates from the master server, and saving it in
> /etc/ipa/ca.crt in this process at the stage Configuring the web
> interface (httpd)  we got the below error i.e.
>
>
>
> ipa-replica-install command failed, exception: CalledProcessError:
> Command '/usr/bin/certutil -d dbm:/etc/httpd/alias -A -n Server-Cert -t
> ,, -a -f /etc/httpd/alias/pwdfile.txt' returned non-zero exit status 255
>
>
>
> ===============================================
>
>
>
> While installing Replica /var/log/ipaclient-install.log
>
> ---------------------------------------------------
>
>
>
> 2022-08-15T13:52:08Z DEBUG stderr=
>
> 2022-08-15T13:52:08Z DEBUG trying to retrieve CA cert via LDAP from
> aaa01.ipa.subdomain.com
>
> 2022-08-15T13:52:09Z DEBUG retrieving schema for SchemaCache
> url=ldap://aaa01.ipa.subdomain.com:389
> conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f17fe812440>
>
> 2022-08-15T13:52:11Z INFO Successfully retrieved CA cert
>
>
>
>     Subject:     CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
>     Issuer:      CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
>     Valid From:  2018-04-12 14:15:30
>
>     Valid Until: 2038-04-12 14:15:30
>
>
>
>     Subject:     CN=dirus02.ipa.subdomain.com,O=IPA.SUBDOMAIN.COM
>
>     Issuer:      CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
>     Valid From:  2019-01-21 11:54:13
>
>     Valid Until: 2021-01-21 11:54:13
>
>
>
> 2022-08-15T13:52:11Z DEBUG Starting external process
>
> 2022-08-15T13:52:11Z DEBUG args=/usr/sbin/ipa-join -s
> aaa01.ipa.subdomain.com -b dc=ipa,dc=example,dc=com -h
> dirpav01-tfln-mdr1-omes.ipa.subdomain.com
>
> 2022-08-15T13:52:15Z DEBUG Process finished, return code=0
>
> 2022-08-15T13:52:15Z DEBUG stdout=
>
> 2022-08-15T13:52:15Z DEBUG stderr=Keytab successfully retrieved and
> stored in: /etc/krb5.keytab
>
> Certificate subject base is: O=IPA.SUBDOMAIN.COM
>
>
>
> 2022-08-15T13:52:15Z INFO Enrolled in IPA realm IPA.SUBDOMAIN.COM
>
> 2022-08-15T13:52:15Z DEBUG Starting external process
>
> 2022-08-15T13:52:15Z DEBUG args=/usr/bin/kdestroy
>
> 2022-08-15T13:52:15Z DEBUG Process finished, return code=0
>
> 2022-08-15T13:52:15Z DEBUG stdout=
>
>
>
> ==================================
>
>
>
>
>
>
>
> While installing replica  /var/log/ipareplica-install.log
>
> --------------------------------------------------
>
>
>
> 2022-08-15T15:07:11Z DEBUG   [14/22]: importing CA certificates from LDAP
>
> 2022-08-15T15:07:11Z DEBUG Loading Index file from
> '/var/lib/ipa/sysrestore/sysrestore.index'
>
> 2022-08-15T15:07:11Z DEBUG Starting external process
>
> 2022-08-15T15:07:11Z DEBUG args=/usr/bin/certutil -d
> dbm:/etc/httpd/alias -A -n IPA.SUBDOMAIN.COM IPA CA -t CT,C,C -a -f
> /etc/httpd/alias/pwdfile.txt
>
> 2022-08-15T15:07:11Z DEBUG Process finished, return code=0
>
> 2022-08-15T15:07:11Z DEBUG stdout=
>
> 2022-08-15T15:07:11Z DEBUG stderr=
>
> 2022-08-15T15:07:11Z DEBUG Starting external process
>
> 2022-08-15T15:07:11Z DEBUG args=/usr/bin/certutil -d
> dbm:/etc/httpd/alias -A -n Server-Cert -t ,, -a -f
> /etc/httpd/alias/pwdfile.txt
>
> 2022-08-15T15:07:12Z DEBUG Process finished, return code=255
>
> 2022-08-15T15:07:12Z DEBUG stdout=
>
> 2022-08-15T15:07:12Z DEBUG stderr=certutil: could not add certificate to
> token or database: SEC_ERROR_ADDING_CERT: Error adding certificate to
> database.
>
>
>
> 2022-08-15T15:07:12Z DEBUG Traceback (most recent call last):
>
>   File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> line 567, in start_creation
>
>     run_step(full_msg, method)
>
>   File "/usr/lib/python2.7/site-packages/ipaserver/install/service.py",
> line 557, in run_step
>
>
>
>
>
> Observation in Master server(aaa01)  ldap database :
>
> =======================================
>
>
>
> [root@aaa01~]# ldapsearch -D 'cn=directory manager' -w XXXXXXXXX |
> grep    "ipaCertSubject"
>
> ipaCertSubject: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
> ipaCertSubject: CN=dirus02.ipa.subdomain.com,O=IPA.SUBDOMAIN.COM
>
> [root@aaa01~]#
>
>
>
> ====================
>
> We could see this certificate
> "CN=dirus02.ipa.subdomain.com,O=IPA.SUBDOMAIN.COM" in IPA master server
> GUI as well we have revoked it too , but still it retrieves the same
> and  installation got fails everytime
>
>
>
> =================
>
>
>
> In ideal case while installing replica it has to retrieve only one
> certificate i.e. CN=Certificate Authority,O=IPA.SUBDOMAIN.COM but this
> case it  retrieves
>
>
>
>
>
> Please let us know if any more details required and let us know how can
> we fix this issue, without impact on whole setup
>
>
>
>
>
> ipaCertIssuerSerial
>
>
>
> ipaCertIssuerSerial: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM;1
> [which is a valid certificate]
>
> ipaCertIssuerSerial: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM;32  [
> invalid certificate retrieves from ipa master while installing ipa replica]
>
>
>
>
>
>
>
> [root@aaa01]# ipa cert-show
>
>
>
> Serial number: 32
>
>   Issuing CA: ipa
>
>   Certificate:
> MIIFGTCCBAGgAwIBAgIBIDANBgkqhkiG9w0BAQsFADA7MRkwFwYDVQQKDBBJUEEuT05NT0JJTEUuQ09NMR4wHAYDVQQ
>
> DDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMTkwMTIxMTE1NDEzWhcNMjEwMTIxMTE1NDEzWjBMMRkwFwYDVQQKDBBJUEEuT
>
> 05NT0JJTEUuQ09NMS8wLQYDVQQDDCZkaXJ1czAyLW1pYS10bGZuLW9tdXMuaXBhLm9ubW9iaWxlLmNvbTCCASIwDQYJKoZIhvcNAQE
>
> BBQADggEPADCCAQoCggEBAKln0qNlB+38cXbyOurkVgK+GMYM9loUVFAvZGlydXMwMi1taWEtdGxmbi1vbXVzLmlwYS5vbm1vYmlsZS5
>
> jb21ASVBBLk9OTU9CSUxFLkNPTaBbBgYrBgEFAgKgUTBPoBIbEElQQS5PTk1PQklMRS5DT02hOTA3oAMCAQGhMDAuGwRIVFRQGyZkaXJ
>
> 1czAyLW1pYS10bGZuLW9tdXMuaXBhLm9ubW9iaWxlLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAcFbSY4tVpZHWVDGsahRNfCqv/x/xCT
>
> BEYHvCSdycHAV7Ogq6zEENviRDOEOYqe1x7BxyF7B/hhB3PX2uqYmFrgPffyfwCxGZb0DRnnOLnwldxe3QdwjIIuUptY9fOgvbjx+bd5iLIgNp
>
> aAZcN70PePdPA0xYpAo3CQkowCojAke2QGsPp6DrXS1wRrE4maH0LmEtu56hSbARoN4DgJ91PKgPkZ+BNyq9BmoPTRsxpAGBvms2SAbx
>
> q1iUmNcVCurqvF/Gu2Z8L5rlpPiVjSbup9Zq5LuhLtfeMsgrwfZOcwZQfSCCykMUH9eAipvsNoHvPxiJeHhDk8Zx+cADESTL4w==
>
>
>
>  Subject: CN=dirus02.ipa.subdomain.com,O=IPA.SUBDOMAIN.COM
>
>   Subject DNS name: dirus02.ipa.subdomain.com
>
>   Subject UPN: HTTP/dirus02.ipa.subdomain.com@IPA.SUBDOMAIN.COM
>
>   Subject Kerberos principal name:
> HTTP/dirus02.ipa.subdomain.com@IPA.SUBDOMAIN.COM
>
>   Issuer: CN=Certificate Authority,O=IPA.SUBDOMAIN.COM
>
>   Not Before: Mon Jan 21 11:54:13 2019 UTC
>
>   Not After: Thu Jan 21 11:54:13 2021 UTC
>
>   Serial number: 32
>
>   Serial number (hex): 0x20
>
>   Revoked: True
>
>   Revocation reason: 2
>
> [root@aaa01~]#

The CA certificates are stored in LDAP under
cn=certificates,cn=ipa,cn=etc,dc=example,dc=test (substitute your own
basedn).

Find the incorrect entry and use ldapdelete to remove it. If you aren't
very familiar with LDAP command-line tools then something like Apache
Directory Studio may be a better choice.

rob



_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue

 




DISCLAIMER: The information in this message is confidential and may be legally privileged. It is intended solely for the addressee. Access to this message by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, or distribution of the message, or any action or omission taken by you in reliance on it, is prohibited and may be unlawful. Please immediately contact the sender if you have received this message in error. Further, this e-mail may contain viruses and all reasonable precaution to minimize the risk arising there from is taken by OnMobile. OnMobile is not liable for any damage sustained by you as a result of any virus in this e-mail. All applicable virus checks should be carried out by you before opening this e-mail or any attachment thereto.
Thank you - OnMobile Global Limited.