I cleaned up the contents of our ldap manually, re-created the replica file, and got a lot further than we have before but ipa-replica-install still failed as below:
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes [1/30]: configuring certificate server instance ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmphvJyRY' returned non-zero exit status 1 ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information: ipaserver.install.dogtaginstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR CA configuration failed. ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information [root@ipa2c7 ~]# ipa-server-install --uninstall -U ipapython.admintool: ERROR Unable to read /etc/httpd/conf.d/ipa-pki-proxy.conf ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information [root@ipa2c7 ~]# touch /etc/httpd/conf.d/ipa-pki-proxy.conf [root@ipa2c7 ~]# ipa-server-install --uninstall -U Deleting this server will leave your installation without a CRL generation master. ipapython.admintool: ERROR Aborting uninstall operation. ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
It looks to have failed in such a way that it doesn't know how to back out again, which I haven't seen before. Thoughts? The error in ipa-uninstall.log looks like a generic admintool.py error:
2021-06-07T12:31:38Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-OUR-NET.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f05289fdf80> 2021-06-07T12:31:38Z DEBUG raw: config_show(version=u'2.237') 2021-06-07T12:31:38Z DEBUG config_show(rights=False, all=False, raw=False, version=u'2.237') 2021-06-07T12:31:38Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run return cfgr.run() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 358, in run self.validate() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in validate for _nothing in self._validator(): File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 633, in _configure next(validator) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 73, in _uninstall for unused in self._uninstaller(self.parent): File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 594, in main uninstall_check(self) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 250, in decorated func(installer) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 1030, in uninstall_check ca.uninstall_check(options) File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 127, in uninstall_check raise ScriptError("Aborting uninstall operation.")
2021-06-07T12:31:38Z DEBUG The ipa-server-install command failed, exception: ScriptError: Aborting uninstall operation. 2021-06-07T12:31:38Z ERROR Aborting uninstall operation. 2021-06-07T12:31:38Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
Bret Wortman wrote:
I cleaned up the contents of our ldap manually, re-created the replica file, and got a lot further than we have before but ipa-replica-install still failed as below:
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes [1/30]: configuring certificate server instance ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmphvJyRY' returned non-zero exit status 1 ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information: ipaserver.install.dogtaginstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
If the pkispawn fails you need to dig into the CA logging for any details. It just gives us a pass/fail to IPA.
ipapython.admintool: ERROR CA configuration failed. ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information [root@ipa2c7 ~]# ipa-server-install --uninstall -U ipapython.admintool: ERROR Unable to read /etc/httpd/conf.d/ipa-pki-proxy.conf ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information [root@ipa2c7 ~]# touch /etc/httpd/conf.d/ipa-pki-proxy.conf [root@ipa2c7 ~]# ipa-server-install --uninstall -U Deleting this server will leave your installation without a CRL generation master. ipapython.admintool: ERROR Aborting uninstall operation. ipapython.admintool: ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
This is most likely a red herring. Your current version is old enough that it may not store who the CRL management server is. On the existing server us ipa config-show to find out.
You can add --igore-last-of-role to skip this I believe.
rob
It looks to have failed in such a way that it doesn't know how to back out again, which I haven't seen before. Thoughts? The error in ipa-uninstall.log looks like a generic admintool.py error:
2021-06-07T12:31:38Z DEBUG retrieving schema for SchemaCache url=ldapi://%2fvar%2frun%2fslapd-OUR-NET.socket conn=<ldap.ldapobject.SimpleLDAPObject instance at 0x7f05289fdf80> 2021-06-07T12:31:38Z DEBUG raw: config_show(version=u'2.237') 2021-06-07T12:31:38Z DEBUG config_show(rights=False, all=False, raw=False, version=u'2.237') 2021-06-07T12:31:38Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 178, in execute return_value = self.run() File "/usr/lib/python2.7/site-packages/ipapython/install/cli.py", line 319, in run return cfgr.run() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 358, in run self.validate() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 368, in validate for _nothing in self._validator(): File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 633, in _configure next(validator) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 431, in __runner exc_handler(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 455, in _handle_validate_exception self._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 518, in _handle_exception self.__parent._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 515, in _handle_exception super(ComponentBase, self)._handle_exception(exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 450, in _handle_exception six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 421, in __runner step() File "/usr/lib/python2.7/site-packages/ipapython/install/core.py", line 418, in <lambda> step = lambda: next(self.__gen) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 81, in run_generator_with_yield_from six.reraise(*exc_info) File "/usr/lib/python2.7/site-packages/ipapython/install/util.py", line 59, in run_generator_with_yield_from value = gen.send(prev_value) File "/usr/lib/python2.7/site-packages/ipapython/install/common.py", line 73, in _uninstall for unused in self._uninstaller(self.parent): File "/usr/lib/python2.7/site-packages/ipaserver/install/server/__init__.py", line 594, in main uninstall_check(self) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 250, in decorated func(installer) File "/usr/lib/python2.7/site-packages/ipaserver/install/server/install.py", line 1030, in uninstall_check ca.uninstall_check(options) File "/usr/lib/python2.7/site-packages/ipaserver/install/ca.py", line 127, in uninstall_check raise ScriptError("Aborting uninstall operation.")
2021-06-07T12:31:38Z DEBUG The ipa-server-install command failed, exception: ScriptError: Aborting uninstall operation. 2021-06-07T12:31:38Z ERROR Aborting uninstall operation. 2021-06-07T12:31:38Z ERROR The ipa-server-install command failed. See /var/log/ipaserver-uninstall.log for more information
You were absolutely correct, the flag worked, and the config-show did not show a CRL server at all.
I'll dig into the ca logs next.
I was tailing several logs in /var/log/pki/pki-tomcat/ca/ (debug, system, and transactions) and though the replica installation failed again at the same point, this is what I got from the logs throughout the installation process:
[08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: SecurityDomainResource.getDomainInfo() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: mapping: default [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: required auth methods: [*] [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: anonymous access allowed [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: SecurityDomainResource.getDomainInfo() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: No ACL mapping. [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: SecurityDomainResource.getDomainInfo() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: content-type: null [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: accept: [application/json] [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: response format: application/json [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: according to ccMode, authorization for servlet: securitydomain is LDAP based, not XML {1}, use default authz mgr: {2}. [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapBoundConnFactory: init [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapBoundConnFactory:doCloning true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAuthInfo: init() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAuthInfo: init begins [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAuthInfo: init ends [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: init: before makeConnection errorIfDown is false [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: makeConnection: errorIfDown false [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapJssSSLSocket set client auth cert nicknamesubsystemCert cert-pki-ca [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SSL handshake happened [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: Established LDAP connection with SSL client auth to ipa1.our.net:636 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: initializing with mininum 3 and maximum 15 connections to host ipa1.our.net port 636, secure connection, true, authentication type 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: increasing minimum connections by 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: new total available connections 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: new number of connections 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: name: IPA [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: CA [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - cn=ipa1.our.net:443,cn=CAList,ou=Security Domain,o=ipaca [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - objectClass: top [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - host: ipa1.our.net [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SecurePort: 443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SecureAgentPort: 443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SecureAdminPort: 443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SecureEEClientAuthPort: 443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - UnSecurePort: 80 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - Clone: FALSE [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SubsystemName: CA ipa1.our.net 8443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - cn: ipa1.our.net:443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - DomainManager: TRUE [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: OCSP [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: KRA [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: RA [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: TKS [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: TPS [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: Releasing ldap connection [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: Authentication: UID=admin [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAnonConnFactory::getConn [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAnonConnFactory.getConn(): num avail conns now 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SSL handshake happened [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: AccountResource.login() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: mapping: account [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: required auth methods: [passwdUserDBAuthMgr, certUserDBAuthMgr] [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: authentication manager: passwdUserDBAuthMgr [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: access granted [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: AccountResource.login() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: mapping: account.login [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: principal: admin [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: ACL: certServer.ca.account,login [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: checkACLS(): ACLEntry expressions= user="anybody" [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: evaluating expressions: user="anybody" [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: evaluated expression: user="anybody" to be true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: DirAclAuthz: authorization passed [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: access granted [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: AccountResource.login() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: content-type: null [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: accept: [application/json] [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: response format: application/json [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: AccountResource.logout() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: mapping: account [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: required auth methods: [passwdUserDBAuthMgr, certUserDBAuthMgr] [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: authentication manager: passwdUserDBAuthMgr [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: access granted [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: AccountResource.logout() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: mapping: account.logout [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: principal: admin [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: ACL: certServer.ca.account,logout [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: checkACLS(): ACLEntry expressions= user="anybody" [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: evaluating expressions: user="anybody" [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: evaluated expression: user="anybody" to be true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: DirAclAuthz: authorization passed [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: access granted [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: AccountResource.logout() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: content-type: null [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: accept: [application/json] [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: response format: application/json
It again failed at this point:
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes [1/30]: configuring certificate server instance ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmph2SUT4' returned non-zero exit status 1 ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information: ipaserver.install.dogtaginstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR CA configuration failed. ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Is there another way to transfer or duplicate the CA? We are ultimately planning to shut this box down due to its age, and currently it is the only CA, but it seems to be trying to hang on to its job security... ;-)
Bret Wortman via FreeIPA-users wrote:
I was tailing several logs in /var/log/pki/pki-tomcat/ca/ (debug, system, and transactions) and though the replica installation failed again at the same point, this is what I got from the logs throughout the installation process:
This doesn't seem to show any errors. Reading the pki logs can be problematic as it often charges on after an error is encountered so subsequent errors are basically red herrings but I don't see anything wrong here at all, or I'm missing something.
The IPA installer calls pki-spawn <bunch of options> so not much comes back to us. It's a black box. Can you provide the whole debug log, out-of-band is fine too. I'd also suggest looking at the debug log on the existing CA as it may be part of the communication as well.
rob
[08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: SecurityDomainResource.getDomainInfo() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: mapping: default [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: required auth methods: [*] [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: anonymous access allowed [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: SecurityDomainResource.getDomainInfo() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: No ACL mapping. [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: SecurityDomainResource.getDomainInfo() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: content-type: null [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: accept: [application/json] [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: response format: application/json [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: according to ccMode, authorization for servlet: securitydomain is LDAP based, not XML {1}, use default authz mgr: {2}. [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapBoundConnFactory: init [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapBoundConnFactory:doCloning true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAuthInfo: init() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAuthInfo: init begins [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAuthInfo: init ends [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: init: before makeConnection errorIfDown is false [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: makeConnection: errorIfDown false [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapJssSSLSocket set client auth cert nicknamesubsystemCert cert-pki-ca [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SSL handshake happened [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: Established LDAP connection with SSL client auth to ipa1.our.net:636 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: initializing with mininum 3 and maximum 15 connections to host ipa1.our.net port 636, secure connection, true, authentication type 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: increasing minimum connections by 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: new total available connections 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: new number of connections 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: name: IPA [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: CA [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - cn=ipa1.our.net:443,cn=CAList,ou=Security Domain,o=ipaca [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - objectClass: top [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - host: ipa1.our.net [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SecurePort: 443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SecureAgentPort: 443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SecureAdminPort: 443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SecureEEClientAuthPort: 443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - UnSecurePort: 80 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - Clone: FALSE [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - SubsystemName: CA ipa1.our.net 8443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - cn: ipa1.our.net:443 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: - DomainManager: TRUE [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: OCSP [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: KRA [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: RA [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: TKS [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SecurityDomainProcessor: subtype: TPS [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: Releasing ldap connection [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: Authentication: UID=admin [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAnonConnFactory::getConn [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: LdapAnonConnFactory.getConn(): num avail conns now 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: SSL handshake happened [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 2 [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:45][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: In LdapBoundConnFactory::getConn() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: masterConn is connected: true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: getConn: conn is connected true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: getConn: mNumConns now 2 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: returnConn: mNumConns now 3 [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: AccountResource.login() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: mapping: account [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: required auth methods: [passwdUserDBAuthMgr, certUserDBAuthMgr] [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: authentication manager: passwdUserDBAuthMgr [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: access granted [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: AccountResource.login() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: mapping: account.login [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: principal: admin [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: ACL: certServer.ca.account,login [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: checkACLS(): ACLEntry expressions= user="anybody" [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: evaluating expressions: user="anybody" [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: evaluated expression: user="anybody" to be true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: DirAclAuthz: authorization passed [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: access granted [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: AccountResource.login() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: content-type: null [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: accept: [application/json] [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: response format: application/json [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: AccountResource.logout() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: mapping: account [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: required auth methods: [passwdUserDBAuthMgr, certUserDBAuthMgr] [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: authentication manager: passwdUserDBAuthMgr [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: AuthMethodInterceptor: access granted [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: AccountResource.logout() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: mapping: account.logout [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: principal: admin [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: ACL: certServer.ca.account,logout [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: checkACLS(): ACLEntry expressions= user="anybody" [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: evaluating expressions: user="anybody" [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: evaluated expression: user="anybody" to be true [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: DirAclAuthz: authorization passed [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: ACLInterceptor: access granted [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: AccountResource.logout() [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: content-type: null [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: accept: [application/json] [08/Jun/2021:06:35:46][ajp-bio-127.0.0.1-8009-exec-2]: MessageFormatInterceptor: response format: application/json
It again failed at this point:
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes [1/30]: configuring certificate server instance ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance: Command '/usr/sbin/pkispawn -s CA -f /tmp/tmph2SUT4' returned non-zero exit status 1 ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information: ipaserver.install.dogtaginstance: CRITICAL /var/log/pki/pki-tomcat [error] RuntimeError: CA configuration failed. Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR CA configuration failed. ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Is there another way to transfer or duplicate the CA? We are ultimately planning to shut this box down due to its age, and currently it is the only CA, but it seems to be trying to hang on to its job security... ;-)
My misunderstanding, sorry. This is from the existing CA since that's where I thought the problem would be. Okay, going back and looking at the debug log on the new server to see if it's more revealing.
Looks like we're missing an LDAP connection port?
[09/Jun/2021:10:02:54][localhost-startStop-1]: LdapBoundConnFactory: init Property internaldb.ldapconn.port missing value
Full debug log is at https://gist.github.com/wortmanb/7782c5c0c4318c2aec17f2eea589b567
Bret Wortman via FreeIPA-users wrote:
Looks like we're missing an LDAP connection port?
[09/Jun/2021:10:02:54][localhost-startStop-1]: LdapBoundConnFactory: init Property internaldb.ldapconn.port missing value
Full debug log is at https://gist.github.com/wortmanb/7782c5c0c4318c2aec17f2eea589b567
That error is expected (and ignored).
It looks like the failure is here:
[09/Jun/2021:10:02:56][http-bio-8443-exec-3]: ConfigurationUtils: GET https://ipa1.our.net:443/ca/admin/ca/getCertChain javax.ws.rs.ProcessingException: Unable to invoke request ... Caused by: java.io.IOException: SocketException cannot write on socket
So I suppose make sure that communication works but given this is the standard https port it seems like it should just work. Look on ipa1.our.net to see if it got a connection at all, or if it logged some error.
rob
On Wed, Jun 9, 2021, at 2:32 PM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
Looks like we're missing an LDAP connection port?
[09/Jun/2021:10:02:54][localhost-startStop-1]: LdapBoundConnFactory: init Property internaldb.ldapconn.port missing value
Full debug log is at https://gist.github.com/wortmanb/7782c5c0c4318c2aec17f2eea589b567
That error is expected (and ignored).
It looks like the failure is here:
[09/Jun/2021:10:02:56][http-bio-8443-exec-3]: ConfigurationUtils: GET https://ipa1.our.net:443/ca/admin/ca/getCertChain javax.ws.rs.ProcessingException: Unable to invoke request ... Caused by: java.io.IOException: SocketException cannot write on socket
So I suppose make sure that communication works but given this is the standard https port it seems like it should just work. Look on ipa1.our.net to see if it got a connection at all, or if it logged some error.
Nothing in the logs on ipa1.our.net (httpd/error_log or httpd/access_log), so I tried this after the failure:
[root@ipa2c7 ~]# curl -XGET https://ipa1.our.net:443/ca/admin/ca/getCertChain --cacert /etc/ipa/ca.crt
and got:
<?xml version="1.0" encoding="UTF-8" standalone="no"?><XMLResponse><Status>0</Status><ChainBase64>MIIDwg [..snip..] </ChainBase64></XMLResponse>
So the server is responding and the client can get the correct response, at least a few seconds later, it can. I did note that without the --cacert on the curl, I did get an error back but with it, I got an answer.
Bret Wortman wrote:
On Wed, Jun 9, 2021, at 2:32 PM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
Looks like we're missing an LDAP connection port?
[09/Jun/2021:10:02:54][localhost-startStop-1]: LdapBoundConnFactory: init Property internaldb.ldapconn.port missing value
Full debug log is at https://gist.github.com/wortmanb/7782c5c0c4318c2aec17f2eea589b567
That error is expected (and ignored).
It looks like the failure is here:
[09/Jun/2021:10:02:56][http-bio-8443-exec-3]: ConfigurationUtils: GET https://ipa1.our.net:443/ca/admin/ca/getCertChain javax.ws.rs.ProcessingException: Unable to invoke request ... Caused by: java.io.IOException: SocketException cannot write on socket
So I suppose make sure that communication works but given this is the standard https port it seems like it should just work. Look on ipa1.our.net to see if it got a connection at all, or if it logged some error.
Nothing in the logs on ipa1.our.net (httpd/error_log or httpd/access_log), so I tried this after the failure:
[root@ipa2c7 ~]# curl -XGET https://ipa1.our.net:443/ca/admin/ca/getCertChain --cacert /etc/ipa/ca.crt
and got:
<?xml version="1.0" encoding="UTF-8" standalone="no"?><XMLResponse><Status>0</Status><ChainBase64>MIIDwg [..snip..] </ChainBase64></XMLResponse>
So the server is responding and the client can get the correct response, at least a few seconds later, it can. I did note that without the --cacert on the curl, I did get an error back but with it, I got an answer.
So you've run ipa-replica-prepare <host> and then ship that file to <host> right?
At some point we started re-generating the CA certs file (/root/cacert.p12) during preparation. Did we do this in F21? I have no idea.
Can you use pk12util to look at the contents of that file? The password is the initial DM password. Look for expirations, things like that.
# pk12util -l /root/cacert.p12
You can generate a new one but it requires putting passwords into files temporarily.
If you need to generate a new one make a backup of the current, put the passwords in files per below and run this:
# PKCS12Export -d /etc/pki/pki-tomcat/alias/ -p /tmp/nssdbpwd -w /tmp/pk12pwd -o /root/cacert.p12
The NSS db password is in /etc/pki/pki-tomcat/password.conf the value internal.
Otherwise I'm not sure what would generate the socket error except a real network issue. Can you run wireshark on the new server during the install to see what is happening?
rob
On Thu, Jun 10, 2021, at 5:45 PM, Rob Crittenden wrote:
So you've run ipa-replica-prepare <host> and then ship that file to <host> right?
Exactly.
At some point we started re-generating the CA certs file (/root/cacert.p12) during preparation. Did we do this in F21? I have no idea.
Can you use pk12util to look at the contents of that file? The password is the initial DM password. Look for expirations, things like that.
# pk12util -l /root/cacert.p12
All the "Not After" dates were in 2022 or 2034, and the "Not Before" dates were all before 2020. So I that all seems fine.
You can generate a new one but it requires putting passwords into files temporarily.
If you need to generate a new one make a backup of the current, put the passwords in files per below and run this:
# PKCS12Export -d /etc/pki/pki-tomcat/alias/ -p /tmp/nssdbpwd -w /tmp/pk12pwd -o /root/cacert.p12
The NSS db password is in /etc/pki/pki-tomcat/password.conf the value internal.
Otherwise I'm not sure what would generate the socket error except a real network issue. Can you run wireshark on the new server during the install to see what is happening?
I could, but these two systems are both VM guests on the same VMware server, on the same virtual subnet. But I will take a deep dive today into the network and see if I can find anything there.
rob
This appears to be the error, or at least it's the only "fatal" I could find in the stream and it's near enough to the end of traffic that it seems likely. I'm no expert on Wireshark so I'm hoping someone is willing to take a peek and let me know if there's something obvious here.
https://gist.github.com/wortmanb/d3b1cb38e894d1fb0578ab05e459b178
Bret Wortman via FreeIPA-users wrote:
This appears to be the error, or at least it's the only "fatal" I could find in the stream and it's near enough to the end of traffic that it seems likely. I'm no expert on Wireshark so I'm hoping someone is willing to take a peek and let me know if there's something obvious here.
https://gist.github.com/wortmanb/d3b1cb38e894d1fb0578ab05e459b178
Are you sure you aren't seeing a connect error on the F21 Apache server? This looks to me like an untrusted CA or something like it.
Have you replaced any of your IPA certs on the F21 server? Signed the IPA CA with an external?
rob
On Mon, Jun 14, 2021, at 3:47 PM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
This appears to be the error, or at least it's the only "fatal" I could find in the stream and it's near enough to the end of traffic that it seems likely. I'm no expert on Wireshark so I'm hoping someone is willing to take a peek and let me know if there's something obvious here.
https://gist.github.com/wortmanb/d3b1cb38e894d1fb0578ab05e459b178
Are you sure you aren't seeing a connect error on the F21 Apache server? This looks to me like an untrusted CA or something like it.
Not that I'm aware of. We haven't touched those servers in ages (hence the F21). Where would we be most likely to see the connect error on the server? I may have missed a log file.
Have you replaced any of your IPA certs on the F21 server? Signed the IPA CA with an external?
I'll double-check today but not that I'm aware of.
On Tue, Jun 15, 2021, at 5:47 AM, Bret Wortman via FreeIPA-users wrote:
On Mon, Jun 14, 2021, at 3:47 PM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
This appears to be the error, or at least it's the only "fatal" I could find in the stream and it's near enough to the end of traffic that it seems likely. I'm no expert on Wireshark so I'm hoping someone is willing to take a peek and let me know if there's something obvious here.
https://gist.github.com/wortmanb/d3b1cb38e894d1fb0578ab05e459b178
Are you sure you aren't seeing a connect error on the F21 Apache server? This looks to me like an untrusted CA or something like it.
Not that I'm aware of. We haven't touched those servers in ages (hence the F21). Where would we be most likely to see the connect error on the server? I may have missed a log file.
Bingo!
192.168.2.215 - - [17/Jun/2021:07:11:28 -0400] "GET /ca/rest/securityDomain/domainInfo HTTP/1.1" 200 190 192.168.2.215 - - [17/Jun/2021:07:11:28 -0400] "GET /ca/rest/account/login HTTP/1.1" 200 188 192.168.2.215 - - [17/Jun/2021:07:11:30 -0400] "GET /ca/rest/account/logout HTTP/1.1" 204 - [Thu Jun 17 07:11:41.806659 2021] [:error] [pid 921] SSL Library Error: -12286 No common encryption algorithm(s) with client
I don't think we adjusted the SSL configs on either end...
Have you replaced any of your IPA certs on the F21 server? Signed the IPA CA with an external?
I'll double-check today but not that I'm aware of. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Jun 17, 2021, at 7:15 AM, Bret Wortman via FreeIPA-users wrote:
On Tue, Jun 15, 2021, at 5:47 AM, Bret Wortman via FreeIPA-users wrote:
On Mon, Jun 14, 2021, at 3:47 PM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
This appears to be the error, or at least it's the only "fatal" I could find in the stream and it's near enough to the end of traffic that it seems likely. I'm no expert on Wireshark so I'm hoping someone is willing to take a peek and let me know if there's something obvious here.
https://gist.github.com/wortmanb/d3b1cb38e894d1fb0578ab05e459b178
Are you sure you aren't seeing a connect error on the F21 Apache server? This looks to me like an untrusted CA or something like it.
Not that I'm aware of. We haven't touched those servers in ages (hence the F21). Where would we be most likely to see the connect error on the server? I may have missed a log file.
Bingo!
192.168.2.215 - - [17/Jun/2021:07:11:28 -0400] "GET /ca/rest/securityDomain/domainInfo HTTP/1.1" 200 190 192.168.2.215 - - [17/Jun/2021:07:11:28 -0400] "GET /ca/rest/account/login HTTP/1.1" 200 188 192.168.2.215 - - [17/Jun/2021:07:11:30 -0400] "GET /ca/rest/account/logout HTTP/1.1" 204 - [Thu Jun 17 07:11:41.806659 2021] [:error] [pid 921] SSL Library Error: -12286 No common encryption algorithm(s) with client
I don't think we adjusted the SSL configs on either end...
So I took the cypher list from the new box and copied it to the other and added it to httpd/conf.d/nss.conf and then the two ends could talk again. We got as far as this now:
Done configuring certificate server (pki-tomcatd). Applying LDAP updates Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: disabling Schema Compat [6/10]: starting directory server [7/10]: upgrading server [8/10]: stopping directory server [9/10]: restoring configuration [10/10]: starting directory server Done. Finalize replication settings Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR Server is unwilling to perform: modification of attribute nsds5ReplicaReleaseTimeout is not allowed in replica entry ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Is there a simple workaround for this?
Have you replaced any of your IPA certs on the F21 server? Signed the IPA CA with an external?
I'll double-check today but not that I'm aware of. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Jun 17, 2021, at 9:54 AM, Bret Wortman via FreeIPA-users wrote:
On Thu, Jun 17, 2021, at 7:15 AM, Bret Wortman via FreeIPA-users wrote:
On Tue, Jun 15, 2021, at 5:47 AM, Bret Wortman via FreeIPA-users wrote:
On Mon, Jun 14, 2021, at 3:47 PM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
This appears to be the error, or at least it's the only "fatal" I could find in the stream and it's near enough to the end of traffic that it seems likely. I'm no expert on Wireshark so I'm hoping someone is willing to take a peek and let me know if there's something obvious here.
https://gist.github.com/wortmanb/d3b1cb38e894d1fb0578ab05e459b178
Are you sure you aren't seeing a connect error on the F21 Apache server? This looks to me like an untrusted CA or something like it.
Not that I'm aware of. We haven't touched those servers in ages (hence the F21). Where would we be most likely to see the connect error on the server? I may have missed a log file.
Bingo!
192.168.2.215 - - [17/Jun/2021:07:11:28 -0400] "GET /ca/rest/securityDomain/domainInfo HTTP/1.1" 200 190 192.168.2.215 - - [17/Jun/2021:07:11:28 -0400] "GET /ca/rest/account/login HTTP/1.1" 200 188 192.168.2.215 - - [17/Jun/2021:07:11:30 -0400] "GET /ca/rest/account/logout HTTP/1.1" 204 - [Thu Jun 17 07:11:41.806659 2021] [:error] [pid 921] SSL Library Error: -12286 No common encryption algorithm(s) with client
I don't think we adjusted the SSL configs on either end...
So I took the cypher list from the new box and copied it to the other and added it to httpd/conf.d/nss.conf and then the two ends could talk again. We got as far as this now:
Done configuring certificate server (pki-tomcatd). Applying LDAP updates Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: disabling Schema Compat [6/10]: starting directory server [7/10]: upgrading server [8/10]: stopping directory server [9/10]: restoring configuration [10/10]: starting directory server Done. Finalize replication settings Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR Server is unwilling to perform: modification of attribute nsds5ReplicaReleaseTimeout is not allowed in replica entry ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Is there a simple workaround for this?
In my Googling for an answer I found some earlier threads that might never have been resolved, but Florence asked some early questions that I thought I'd answer right now: On ipa1 (original F21 server):
# rpm -qa | grep 389 389-ds-base-libs-1.3.3.13-1.fc21.x86_64 389-ds-base-1.3.3.13-1.fc21.x86_64 # ldapsearch -Y GSSAPI -h ipa1 -b cn=schema -s base -o ldif-wrap=no -LLL attributetypes | grep -i nsds5replicareleasetimeout SASL/GSSAPI authentication started SASL username: admin@OUR.NET SASL SSF: 56 SASL data security layer installed. attributetypes: ( 2.16.840.1.113730.3.1.2333 NAME 'nsds5ReplicaReleaseTimeout' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE X-ORIGIN ( 'Netscape Directory Server' 'user defined' ) )
On ipa2c7 (new C7 server we're trying to add as a replica):
# rpm -qa | grep 389 389-ds-base-libs-1.3.10.2-10.el7_9.x86_64 389-ds-base-1.3.10.2-10.el7_9.x86_64 # ldapsearch -Y GSSAPI -h ipa1 -b cn=schema -s base -o ldif-wrap=no -LLL attributetypes | grep -i nsds5replicareleasetimeout SASL/GSSAPI authentication started SASL username: host/ipa2c7.wedgeofli.me@OUR.NET SASL SSF: 256 SASL data security layer installed. attributetypes: ( 2.16.840.1.113730.3.1.2333 NAME 'nsds5ReplicaReleaseTimeout' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE X-ORIGIN ( 'Netscape Directory Server' 'user defined' ) )
Have you replaced any of your IPA certs on the F21 server? Signed the IPA CA with an external?
I'll double-check today but not that I'm aware of. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Bret Wortman via FreeIPA-users wrote:
On Thu, Jun 17, 2021, at 9:54 AM, Bret Wortman via FreeIPA-users wrote:
On Thu, Jun 17, 2021, at 7:15 AM, Bret Wortman via FreeIPA-users wrote:
On Tue, Jun 15, 2021, at 5:47 AM, Bret Wortman via FreeIPA-users wrote:
On Mon, Jun 14, 2021, at 3:47 PM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
This appears to be the error, or at least it's the only "fatal" I could find in the stream and it's near enough to the end of traffic that it seems likely. I'm no expert on Wireshark so I'm hoping someone is willing to take a peek and let me know if there's something obvious here.
https://gist.github.com/wortmanb/d3b1cb38e894d1fb0578ab05e459b178
Are you sure you aren't seeing a connect error on the F21 Apache server? This looks to me like an untrusted CA or something like it.
Not that I'm aware of. We haven't touched those servers in ages (hence the F21). Where would we be most likely to see the connect error on the server? I may have missed a log file.
Bingo!
192.168.2.215 - - [17/Jun/2021:07:11:28 -0400] "GET /ca/rest/securityDomain/domainInfo HTTP/1.1" 200 190 192.168.2.215 - - [17/Jun/2021:07:11:28 -0400] "GET /ca/rest/account/login HTTP/1.1" 200 188 192.168.2.215 - - [17/Jun/2021:07:11:30 -0400] "GET /ca/rest/account/logout HTTP/1.1" 204 - [Thu Jun 17 07:11:41.806659 2021] [:error] [pid 921] SSL Library Error: -12286 No common encryption algorithm(s) with client
I don't think we adjusted the SSL configs on either end...
So I took the cypher list from the new box and copied it to the other and added it to httpd/conf.d/nss.conf and then the two ends could talk again. We got as far as this now:
Done configuring certificate server (pki-tomcatd). Applying LDAP updates Upgrading IPA:. Estimated time: 1 minute 30 seconds [1/10]: stopping directory server [2/10]: saving configuration [3/10]: disabling listeners [4/10]: enabling DS global lock [5/10]: disabling Schema Compat [6/10]: starting directory server [7/10]: upgrading server [8/10]: stopping directory server [9/10]: restoring configuration [10/10]: starting directory server Done. Finalize replication settings Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR Server is unwilling to perform: modification of attribute nsds5ReplicaReleaseTimeout is not allowed in replica entry ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
Is there a simple workaround for this?
I think it will involve editing code on the C7 server.
/usr/lib/python2.7/site-packages/ipaserver/install/replication.py
REPLICA_CREATION_SETTINGS and REPLICA_FINAL_SETTINGS.
Remove the nsds5ReplicaReleaseTimeout from both then try the install again.
When (hopefully) the install is succesfully restore the code, either by manually adding it back in or re-installing the python2-ipaserver package (preferred as it will verify correctly with rpm).
If you want to read about the reasoning of the change see https://pagure.io/freeipa/issue/7617 and the associated PR.
rob
In my Googling for an answer I found some earlier threads that might never have been resolved, but Florence asked some early questions that I thought I'd answer right now: On ipa1 (original F21 server):
# rpm -qa | grep 389 389-ds-base-libs-1.3.3.13-1.fc21.x86_64 389-ds-base-1.3.3.13-1.fc21.x86_64 # ldapsearch -Y GSSAPI -h ipa1 -b cn=schema -s base -o ldif-wrap=no -LLL attributetypes | grep -i nsds5replicareleasetimeout SASL/GSSAPI authentication started SASL username: admin@OUR.NET SASL SSF: 56 SASL data security layer installed. attributetypes: ( 2.16.840.1.113730.3.1.2333 NAME 'nsds5ReplicaReleaseTimeout' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE X-ORIGIN ( 'Netscape Directory Server' 'user defined' ) )
On ipa2c7 (new C7 server we're trying to add as a replica):
# rpm -qa | grep 389 389-ds-base-libs-1.3.10.2-10.el7_9.x86_64 389-ds-base-1.3.10.2-10.el7_9.x86_64 # ldapsearch -Y GSSAPI -h ipa1 -b cn=schema -s base -o ldif-wrap=no -LLL attributetypes | grep -i nsds5replicareleasetimeout SASL/GSSAPI authentication started SASL username: host/ipa2c7.wedgeofli.me@OUR.NET SASL SSF: 256 SASL data security layer installed. attributetypes: ( 2.16.840.1.113730.3.1.2333 NAME 'nsds5ReplicaReleaseTimeout' DESC 'Netscape defined attribute type' SYNTAX 1.3.6.1.4.1.1466.115.121.1.27 SINGLE-VALUE X-ORIGIN ( 'Netscape Directory Server' 'user defined' ) )
Have you replaced any of your IPA certs on the F21 server? Signed the IPA CA with an external?
I'll double-check today but not that I'm aware of. _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Jun 17, 2021, at 2:07 PM, Rob Crittenden wrote:
I think it will involve editing code on the C7 server.
/usr/lib/python2.7/site-packages/ipaserver/install/replication.py
REPLICA_CREATION_SETTINGS and REPLICA_FINAL_SETTINGS.
Remove the nsds5ReplicaReleaseTimeout from both then try the install again.
That did the trick. Our new ipa2c7 is now a replica alongside the others. I'm going to let everything sit now through the weekend but will rebuild each of the F21 servers on C7 and let them replicate from this new one so we don't have to update any IPs or hostnames in any client configs.
When (hopefully) the install is successfully restore the code, either by manually adding it back in or re-installing the python2-ipaserver package (preferred as it will verify correctly with rpm).
Done.
Thanks again, Rob!
Bret
Bret Wortman wrote:
On Thu, Jun 17, 2021, at 2:07 PM, Rob Crittenden wrote:
I think it will involve editing code on the C7 server.
/usr/lib/python2.7/site-packages/ipaserver/install/replication.py
REPLICA_CREATION_SETTINGS and REPLICA_FINAL_SETTINGS.
Remove the nsds5ReplicaReleaseTimeout from both then try the install again.
That did the trick. Our new ipa2c7 is now a replica alongside the others. I'm going to let everything sit now through the weekend but will rebuild each of the F21 servers on C7 and let them replicate from this new one so we don't have to update any IPs or hostnames in any client configs.
When (hopefully) the install is successfully restore the code, either by manually adding it back in or re-installing the python2-ipaserver package (preferred as it will verify correctly with rpm).
Done.
Thanks again, Rob!
Awesome, glad to hear it. When you complete the migration don't forget to move over the DNA settings, CRL generation and other stuff.
rob
On Fri, Jun 18, 2021, at 1:32 PM, Rob Crittenden wrote:
Awesome, glad to hear it. When you complete the migration don't forget to move over the DNA settings, CRL generation and other stuff.
Is this documented somewhere? I'd hate to miss a step.
Bret
On Mon, Jun 21, 2021, at 9:03 AM, Bret Wortman via FreeIPA-users wrote:
On Fri, Jun 18, 2021, at 1:32 PM, Rob Crittenden wrote:
Awesome, glad to hear it. When you complete the migration don't forget to move over the DNA settings, CRL generation and other stuff.
Is this documented somewhere? I'd hate to miss a step.
Also, my new host, ipa2, is claiming to already have a replication agreement with ipa2c7 but I'm not seeing it:
[root@ipa2c7 ~]# ipa-replica-manage list ipa1.our.net: master ipa2c7.our.net: master [root@ipa2c7 ~]# ipa-replica-manage list-ruv Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} Replica Update Vectors: ipa2c7.our.net:389: 26 ipa1.our.net:389: 4 Certificate Server Replica Update Vectors: ipa2c7.our.net:389: 91 ipa1.our.net:389: 96 [root@ipa2c7 ~]#
Could it be one of those "unable to decode" replicas and if so how do I get rid of those?
Bret Wortman via FreeIPA-users wrote:
On Mon, Jun 21, 2021, at 9:03 AM, Bret Wortman via FreeIPA-users wrote:
On Fri, Jun 18, 2021, at 1:32 PM, Rob Crittenden wrote:
Awesome, glad to hear it. When you complete the migration don't forget to move over the DNA settings, CRL generation and other stuff.
Is this documented somewhere? I'd hate to miss a step.
Also, my new host, ipa2, is claiming to already have a replication agreement with ipa2c7 but I'm not seeing it:
[root@ipa2c7 ~]# ipa-replica-manage list ipa1.our.net: master ipa2c7.our.net: master [root@ipa2c7 ~]# ipa-replica-manage list-ruv Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} Replica Update Vectors: ipa2c7.our.net:389: 26 ipa1.our.net:389: 4 Certificate Server Replica Update Vectors: ipa2c7.our.net:389: 91 ipa1.our.net:389: 96 [root@ipa2c7 ~]#
Could it be one of those "unable to decode" replicas and if so how do I get rid of those?
Try ipa-replica-manage clean-dangling-ruv
and/or ipa-replica-manage clean-ruv <replica id>
so 13, 14, 21, etc.
rob
On Mon, Jun 21, 2021, at 10:55 AM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
On Mon, Jun 21, 2021, at 9:03 AM, Bret Wortman via FreeIPA-users wrote:
On Fri, Jun 18, 2021, at 1:32 PM, Rob Crittenden wrote:
Awesome, glad to hear it. When you complete the migration don't forget to move over the DNA settings, CRL generation and other stuff.
Is this documented somewhere? I'd hate to miss a step.
Also, my new host, ipa2, is claiming to already have a replication agreement with ipa2c7 but I'm not seeing it:
[root@ipa2c7 ~]# ipa-replica-manage list ipa1.our.net: master ipa2c7.our.net: master [root@ipa2c7 ~]# ipa-replica-manage list-ruv Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} Replica Update Vectors: ipa2c7.our.net:389: 26 ipa1.our.net:389: 4 Certificate Server Replica Update Vectors: ipa2c7.our.net:389: 91 ipa1.our.net:389: 96 [root@ipa2c7 ~]#
Could it be one of those "unable to decode" replicas and if so how do I get rid of those?
Try ipa-replica-manage clean-dangling-ruv
and/or ipa-replica-manage clean-ruv <replica id>
I did the clean-dangling-ruv and it got me to this point. When I try to clean-ruv one of these IDs:
[root@ipa2c7 ~]# ipa-replica-manage clean-ruv 13 Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} Replica ID 13 not found [root@ipa2c7 ~]#
And it does the same for each.
On Mon, Jun 21, 2021, at 11:02 AM, Bret Wortman via FreeIPA-users wrote:
On Mon, Jun 21, 2021, at 10:55 AM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
On Mon, Jun 21, 2021, at 9:03 AM, Bret Wortman via FreeIPA-users wrote:
On Fri, Jun 18, 2021, at 1:32 PM, Rob Crittenden wrote:
Awesome, glad to hear it. When you complete the migration don't forget to move over the DNA settings, CRL generation and other stuff.
Is this documented somewhere? I'd hate to miss a step.
Also, my new host, ipa2, is claiming to already have a replication agreement with ipa2c7 but I'm not seeing it:
[root@ipa2c7 ~]# ipa-replica-manage list ipa1.our.net: master ipa2c7.our.net: master [root@ipa2c7 ~]# ipa-replica-manage list-ruv Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} Replica Update Vectors: ipa2c7.our.net:389: 26 ipa1.our.net:389: 4 Certificate Server Replica Update Vectors: ipa2c7.our.net:389: 91 ipa1.our.net:389: 96 [root@ipa2c7 ~]#
Could it be one of those "unable to decode" replicas and if so how do I get rid of those?
Try ipa-replica-manage clean-dangling-ruv
and/or ipa-replica-manage clean-ruv <replica id>
I did the clean-dangling-ruv and it got me to this point. When I try to clean-ruv one of these IDs:
[root@ipa2c7 ~]# ipa-replica-manage clean-ruv 13 Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} Replica ID 13 not found [root@ipa2c7 ~]#
And it does the same for each.
ipa-replica-install (from a file) fails at LDAP each time without exception and I'm at a loss. I assume this is the local LDAP (it's up and running, as is the one on the master). The session running the install shows this:
[28/42]: ignore time skew for initial replication [29/42]: setting up initial replication Starting replication, please wait until this has completed. Update in progress, 14 seconds elapsed [ldap://ipa2c7.our.net:389] reports: Update failed! Status: [Error (-1) - LDAP error: Can't contact LDAP server]
[error] RuntimeError: Failed to start replication Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
ipapython.admintool: ERROR Failed to start replication ipapython.admintool: ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information
While /var/log/dirsrv/slapd-OUR-NET/errors shows:
[21/Jun/2021:16:09:27.828454697 +0000] - NOTICE - ldbm_back_start - found 3880256k physical memory [21/Jun/2021:16:09:27.828686874 +0000] - NOTICE - ldbm_back_start - found 3217604k available [21/Jun/2021:16:09:27.828889226 +0000] - NOTICE - ldbm_back_start - cache autosizing: db cache: 97006k [21/Jun/2021:16:09:27.829111098 +0000] - NOTICE - ldbm_back_start - cache autosizing: userRoot entry cache (1 total): 262144k [21/Jun/2021:16:09:27.830332460 +0000] - NOTICE - ldbm_back_start - cache autosizing: userRoot dn cache (1 total): 65536k [21/Jun/2021:16:09:27.830869767 +0000] - NOTICE - ldbm_back_start - total cache size: 415011962 B; [21/Jun/2021:16:09:27.966802789 +0000] - INFO - slapd_daemon - slapd started. Listening on All Interfaces port 389 for LDAP requests [21/Jun/2021:16:09:27.967189984 +0000] - INFO - slapd_daemon - Listening on /var/run/slapd-OUR-NET.socket for LDAPI requests [21/Jun/2021:16:09:28.106773443 +0000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToipa2c7.our.net" (ipa2c7:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later. [21/Jun/2021:16:09:28.119010503 +0000] - ERR - NSMMReplicationPlugin - acquire_replica - agmt="cn=meToipa2c7.our.net" (ipa2c7:389): Unable to acquire replica: permission denied. The bind dn "" does not have permission to supply replication updates to the replica. Will retry later. [21/Jun/2021:16:09:31.136160660 +0000] - WARN - NSMMReplicationPlugin - repl5_inc_run - agmt="cn=meToipa2c7.our.net" (ipa2c7:389): The remote replica has a different database generation ID than the local database. You may have to reinitialize the remote replica, or the local replica.
I'm now trying to detach ipa2c7 from ipa1, the server from which it was originally replicated in an attempt to use the newer replication mechanism to create a pair of replicas. It appears that we're both connected and disconnected at the same time:
[root@ipa2c7 ~]# ipa-replica-manage del ipa1.our.net 'ipa2c7.our.net' has no replication agreement for 'ipa1.our.net' [root@ipa2c7 ~]# ipa-replica-manage list ipa1.our.net: master ipa2c7.our.net: master [root@ipa2c7 ~]# ipa domainlevel-set 1 ipa: ERROR: Domain Level cannot be raised to 1, server ipa1.our.net does not support it. [root@ipa2c7 ~]#
Bret Wortman via FreeIPA-users wrote:
I'm now trying to detach ipa2c7 from ipa1, the server from which it was originally replicated in an attempt to use the newer replication mechanism to create a pair of replicas. It appears that we're both connected and disconnected at the same time:
[root@ipa2c7 ~]# ipa-replica-manage del ipa1.our.net 'ipa2c7.our.net' has no replication agreement for 'ipa1.our.net' [root@ipa2c7 ~]# ipa-replica-manage list ipa1.our.net: master ipa2c7.our.net: master [root@ipa2c7 ~]# ipa domainlevel-set 1 ipa: ERROR: Domain Level cannot be raised to 1, server ipa1.our.net does not support it. [root@ipa2c7 ~]#
`ipa-replica-manage list` lists all known masters.
Try ipa server-del to remove ipa1.
rob
That worked, and I've got a CLEANALLRUV task running for the remaining RUV between the two.
Now, this morning, I've hit the wall on this yet again.
[root@ipa2c7 ~]# ipa-replica-manage list ipa2c7.our.net: master [root@ipa2c7 ~]# ipa-replica-manage list-ruv Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} 53722a35000000050000 5a11c065000000050000 Replica Update Vectors: ipa2c7.our.net:389: 26 Certificate Server Replica Update Vectors: ipa2c7.our.net:389: 91 [root@ipa2c7 ~]# dsctl slapd-OUR-NET db2ldif --replication userRoot /root/userroot.ldif -bash: dsctl: command not found [root@ipa2c7 ~]# db2ldif -n userRoot -Z OUR-NET -a /root/userroot.ldif Exported ldif file: /root/userroot.ldif ldiffile: /root/userroot.ldif [root@ipa2c7 ~]# ls /root/userroot.ldif ls: cannot access /root/userroot.ldif: No such file or directory [root@ipa2c7 ~]# ipa domainlevel-set 1 ipa: ERROR: Domain Level cannot be raised to 1, existing replication conflicts have to be resolved. [root@ipa2c7 ~]# rpm -qa | grep ipa-server ipa-server-4.6.8-5.el7.centos.6.x86_64 ipa-server-dns-4.6.8-5.el7.centos.6.noarch ipa-server-common-4.6.8-5.el7.centos.6.noarch [root@ipa2c7 ~]# rpm -qa | grep 389 389-ds-base-libs-1.3.10.2-12.el7_9.x86_64 389-ds-base-1.3.10.2-12.el7_9.x86_64 [root@ipa2c7 ~]#
What this seems to tell me is that I need a way to remove those replicas, but when I try them by number I don't get very far:
[root@ipa2c7 ~]# ipa-replica-manage clean-ruv 5 Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} 53722a35000000050000 5a11c065000000050000 Replica ID 5 not found
I'd try interacting with the LDAP directly but I can't get an LDIF using db2ldif either with dsctl or without it (because I don't have dsctl and "yum provides */dsctl" returns no hits so I'm not sure where it comes from).
On Wed, Jun 23, 2021, at 5:27 AM, Bret Wortman via FreeIPA-users wrote:
Now, this morning, I've hit the wall on this yet again.
[root@ipa2c7 ~]# ipa-replica-manage list ipa2c7.our.net: master [root@ipa2c7 ~]# ipa-replica-manage list-ruv Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} 53722a35000000050000 5a11c065000000050000 Replica Update Vectors: ipa2c7.our.net:389: 26 Certificate Server Replica Update Vectors: ipa2c7.our.net:389: 91 [root@ipa2c7 ~]# dsctl slapd-OUR-NET db2ldif --replication userRoot /root/userroot.ldif -bash: dsctl: command not found [root@ipa2c7 ~]# db2ldif -n userRoot -Z OUR-NET -a /root/userroot.ldif Exported ldif file: /root/userroot.ldif ldiffile: /root/userroot.ldif [root@ipa2c7 ~]# ls /root/userroot.ldif ls: cannot access /root/userroot.ldif: No such file or directory [root@ipa2c7 ~]# ipa domainlevel-set 1 ipa: ERROR: Domain Level cannot be raised to 1, existing replication conflicts have to be resolved. [root@ipa2c7 ~]# rpm -qa | grep ipa-server ipa-server-4.6.8-5.el7.centos.6.x86_64 ipa-server-dns-4.6.8-5.el7.centos.6.noarch ipa-server-common-4.6.8-5.el7.centos.6.noarch [root@ipa2c7 ~]# rpm -qa | grep 389 389-ds-base-libs-1.3.10.2-12.el7_9.x86_64 389-ds-base-1.3.10.2-12.el7_9.x86_64 [root@ipa2c7 ~]#
What this seems to tell me is that I need a way to remove those replicas, but when I try them by number I don't get very far:
[root@ipa2c7 ~]# ipa-replica-manage clean-ruv 5 Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} 53722a35000000050000 5a11c065000000050000 Replica ID 5 not found
I'd try interacting with the LDAP directly but I can't get an LDIF using db2ldif either with dsctl or without it (because I don't have dsctl and "yum provides */dsctl" returns no hits so I'm not sure where it comes from).
Scratch this last; it was a permissions error when trying to write the output file. But I'm not sure that's going to help me remove these "unable to decode" replicas.
Bret Wortman via FreeIPA-users wrote:
[root@ipa2c7 ~]# ipa-replica-manage clean-ruv 5 Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} 53722a35000000050000 5a11c065000000050000 Replica ID 5 not found
I'd try interacting with the LDAP directly but I can't get an LDIF using db2ldif either with dsctl or without it (because I don't have dsctl and "yum provides */dsctl" returns no hits so I'm not sure where it comes from).
Scratch this last; it was a permissions error when trying to write the output file. But I'm not sure that's going to help me remove these "unable to decode" replicas.
The RUV search uses this filter against your suffix (dc=our,dc=net):
'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))
nsds50ruv is the attribute which contains the replication information.
In fact, it seems that ipa2c7 doesn't want to delete the replica because it thinks is still has a replication agreement with ipa2, but none of the usual tools let me get in there and remove it.
[root@ipa2c7 slapd-OUR-NET]# ipa-replica-manage del ipa2.our.net Connection to 'ipa2.our.net' failed: cannot connect to
'ldaps://ipa2.our.net:636':
Unable to delete replica 'ipa2.our.net' [root@ipa2c7 slapd-OUR-NET]# ipa server-del ipa2.our.net Removing ipa2.our.net from replication topology, please wait... ipa: ERROR: ipa2.our.net: server not found [root@ipa2c7 slapd-OUR-NET]# [23/Jun/2021:11:37:18.017970634 +0000] -
ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa2.our.net" (ipa2:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[23/Jun/2021:11:37:18.027255085 +0000] - NOTICE -
NSMMReplicationPlugin - CleanAllRUV Task (rid 5): Replica not online (agmt="cn=meToipa2.our.net" (ipa2:389))
[23/Jun/2021:11:37:18.028593839 +0000] - NOTICE -
NSMMReplicationPlugin - CleanAllRUV Task (rid 5): Not all replicas online, retrying in 160 seconds...
Is this, then, a case where I need to go back and use ldapdelete to
remove any entry that has a reference to ipa2.our.net
You can try --force to have the iap-replica-manage delete work without being able to contact the remote server. It may work if you're still at DL0.
rob
On Wed, Jun 23, 2021, at 2:13 PM, Rob Crittenden wrote:
Bret Wortman via FreeIPA-users wrote:
[root@ipa2c7 ~]# ipa-replica-manage clean-ruv 5 Directory Manager password:
unable to decode: {replica 13} 60b907570001000d0000 60b907570001000d0000 unable to decode: {replica 14} 60b923030002000e0000 60b923030002000e0000 unable to decode: {replica 21} 60cb27ed000600150000 60cb27ed000600150000 unable to decode: {replica 24} 60cc5b11000400180000 60cc5b11000400180000 unable to decode: {replica 17} 60be13a5000000110000 60be13c9000700110000 unable to decode: {replica 18} 60bf4aec000000120000 60c07065000200120000 unable to decode: {replica 5} 53722a35000000050000 5a11c065000000050000 Replica ID 5 not found
I'd try interacting with the LDAP directly but I can't get an LDIF using db2ldif either with dsctl or without it (because I don't have dsctl and "yum provides */dsctl" returns no hits so I'm not sure where it comes from).
Scratch this last; it was a permissions error when trying to write the output file. But I'm not sure that's going to help me remove these "unable to decode" replicas.
The RUV search uses this filter against your suffix (dc=our,dc=net):
'(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))
nsds50ruv is the attribute which contains the replication information.
This gave me replica 26, but the ldapmodify doesn't seem to have cleaned it. Perhaps because it looks like it's a replication agreement with itself?
[root@ipa2c7 ~]# ldapsearch -xLLL -D "cn=directory manager" -W -s sub -b "dc=our,dc=net" '(&(nsuniqueid=ffffffff-ffffffff-ffffffff-ffffffff)(objectclass=nstombstone))' Enter LDAP Password: dn: cn=replica,cn=dc\3Dour\2Cdc\3Dnet,cn=mapping tree,cn=config cn: replica nsDS5Flags: 1 nsDS5ReplicaBindDN: cn=replication manager,cn=config nsDS5ReplicaBindDnGroupCheckInterval: 2 nsDS5ReplicaId: 26 nsDS5ReplicaName: 2ce04b96-d02b11eb-ad3cb71e-5a893ec1 nsDS5ReplicaRoot: dc=our,dc=net nsDS5ReplicaType: 3 nsState:: GgAAAAAAAABeXNRgAAAAAAAAAAAAAAAAAQMAAAAAAAACAAAAAAAAAA== nsds5ReplicaBackoffMax: 3 nsds5ReplicaLegacyConsumer: off nsds5ReplicaReleaseTimeout: 20 nsds5replicabinddngroup: cn=replication managers,cn=sysaccounts,cn=etc,dc=wedg eofli,dc=net objectClass: top objectClass: nsds5replica objectClass: extensibleobject nsds5ReplicaCleanRUV: 4:60d21f1b000100040000:no:0 nsds50ruv: {replicageneration} 537228a6000000040000 nsds50ruv: {replica 26 ldap://ipa2c7.our.net:389} 60cc8b3c0002001a0000 60 d45f5f0001001a0000 nsruvReplicaLastModified: {replica 26 ldap://ipa2c7.our.net:389} 60d45c5e nsds5ReplicaChangeCount: 14571 nsds5replicareapactive: 0 ldapmodify -x -D "cn=directory manager" -W <<EOF dn: cn=replica,cn=dc\3Dour\2Cdc\3Dnet,cn=mapping tree,cn=config changetype: modify replace: nsds5task nsds5task: CLEANRUV26 EOF [root@ipa2c7 ~]# sleep 120 [root@ipa2c7 ~]# ipa-replica-manage list-ruv Directory Manager password:
unable to decode: {replica 97} 60cb52d2000300610000 60cb5311000e00610000 Replica Update Vectors: ipa2c7.our.net:389: 26 Certificate Server Replica Update Vectors: ipa2c7.our.net:389: 91 ipa1.our.net:389: 96 [root@ipa2c7 ~]#
In fact, it seems that ipa2c7 doesn't want to delete the replica because it thinks is still has a replication agreement with ipa2, but none of the usual tools let me get in there and remove it.
[root@ipa2c7 slapd-OUR-NET]# ipa-replica-manage del ipa2.our.net Connection to 'ipa2.our.net' failed: cannot connect to
'ldaps://ipa2.our.net:636':
Unable to delete replica 'ipa2.our.net' [root@ipa2c7 slapd-OUR-NET]# ipa server-del ipa2.our.net Removing ipa2.our.net from replication topology, please wait... ipa: ERROR: ipa2.our.net: server not found [root@ipa2c7 slapd-OUR-NET]# [23/Jun/2021:11:37:18.017970634 +0000] -
ERR - NSMMReplicationPlugin - bind_and_check_pwp - agmt="cn=meToipa2.our.net" (ipa2:389) - Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[23/Jun/2021:11:37:18.027255085 +0000] - NOTICE -
NSMMReplicationPlugin - CleanAllRUV Task (rid 5): Replica not online (agmt="cn=meToipa2.our.net" (ipa2:389))
[23/Jun/2021:11:37:18.028593839 +0000] - NOTICE -
NSMMReplicationPlugin - CleanAllRUV Task (rid 5): Not all replicas online, retrying in 160 seconds...
Is this, then, a case where I need to go back and use ldapdelete to
remove any entry that has a reference to ipa2.our.net
You can try --force to have the iap-replica-manage delete work without being able to contact the remote server. It may work if you're still at DL0.
--force didn't help much, but the cleanallruv.pl script seems to have helped, once I found it. I fed it all the replica numbers above one by one and now I'm down to the list above.
I've tried using that script and the ldapmodify command above on these replica IDs but they're all stubbornly refusing to be deleted.
I'm considering biting the bullet and just starting over. I've got a dump of our user list and our DNS, and can rebuild from that in about a day if I have to.
Bret Wortman wrote:
On Fri, Jun 18, 2021, at 1:32 PM, Rob Crittenden wrote:
Awesome, glad to hear it. When you complete the migration don't forget to move over the DNA settings, CRL generation and other stuff.
Is this documented somewhere? I'd hate to miss a step.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/htm...
freeipa-users@lists.fedorahosted.org
-
Bret Wortman -
Rob Crittenden