Situation: Two servers housed in 2 different AWS regions are completely
disconnected and totally out of sync. I can't fix replication at all so
I'm looking for clues or tips ..
Backstory:
- Complex Active Directory needs including transitive trusts across
multiple child domains of the AD Forrest
- This means that we've been constantly upgrading IPA and subsystems
like sssd* given the speed at which AD integration is being improved/fixed
- We've been doing "yum update" and "ipa-server-upgrade"
commands all
the way from ipa-3.x to current v4.4.0
- Due to incremental upgrades over time we've been at "domain level 0"
until very recently
Issues
- Two servers work but they are islands to their own - no replication
seems to be occurring
- IPA connection-check scripts seem to all pass
- IPA replication-manage "list" commands seem to work fine
- forcing replication or forcing a complete reinit has zero effect
- IPA topologysegment-find domain commands seem to show the proper
segments
- BUT -- the topology-verify command clearly shows broken topology and
disconnected state
It was only recently that I discovered the broken topology status - had
spent too much time in the weeds looking at debug output trying to
figure out why replication was not working .
I'm wondering what the best next-step is to regaining a unified IPA
view. From reading the admin guide I'm thinking that I need to bring up
new IPA servers so that I have more "nodes" to play with when
potentially connecting and fixing the topology segments -- seems easier
to fix segments when you have more nodes to play with.
I'm not sure what to fix first -- is the broken topology segment the
cause for broken replication or is something wrong in the replication
internals that results in a disconnected topology?
Guidance appreciated. I'm appending some redacted command output below.
Regards,
Chris
###
# ipa-replica-manage list
us-idmp001.COMPANYidm.org: master
eu-idmp001.COMPANYidm.org: master
# ipa topologysegment-find domain
-----------------
1 segment matched
-----------------
Segment name:
us-idmp001.COMPANYidm.org-to-eu-idmp001.COMPANYidm.org
Left node:
us-idmp001.COMPANYidm.org
Right node:
eu-idmp001.COMPANYidm.org
Connectivity: left-right
----------------------------
Number of entries returned 1
----------------------------
#ipa topologysegment-find domain
-----------------
1 segment matched
-----------------
Segment name:
eu-idmp001.COMPANYidm.org-to-us-idmp001.COMPANYidm.org
Left node:
eu-idmp001.COMPANYidm.org
Right node:
us-idmp001.COMPANYidm.org
Connectivity: left-right
----------------------------
Number of entries returned 1
----------------------------
[root@eu-idmp001 centos]#
# ipa topologysuffix-verify domain
========================================================
Replication topology of suffix "domain" contains errors.
========================================================
------------------------
Topology is disconnected
------------------------
Server
eu-idmp001.COMPANYidm.org can't contact servers:
us-idmp001.COMPANYidm.org
[root@us-idmp001 centos]#
# ipa topologysuffix-verify domain
========================================================
Replication topology of suffix "domain" contains errors.
========================================================
------------------------
Topology is disconnected
------------------------
Server
us-idmp001.COMPANYidm.org can't contact servers:
eu-idmp001.COMPANYidm.org
[root@eu-idmp001 centos]#
[root@eu-idmp001 centos]#
# /usr/sbin/ipa-replica-conncheck --replica
eu-idmp001.COMPANYidm.org
Check connection from master to remote replica 'eu-idmp001.COMPANYidm.org':
Directory Service: Unsecure port (389): OK
Directory Service: Secure port (636): OK
Kerberos KDC: TCP (88): OK
Kerberos KDC: UDP (88): WARNING
Kerberos Kpasswd: TCP (464): OK
Kerberos Kpasswd: UDP (464): WARNING
HTTP Server: Unsecure port (80): OK
HTTP Server: Secure port (443): OK
The following UDP ports could not be verified as open: 88, 464
This can happen if they are already bound to an application
and ipa-replica-conncheck cannot attach own UDP responder.
Connection from master to replica is OK.
ipa-replica-conncheck --master
us-idmp001.COMPANYidm.org
Check connection from replica to remote master 'us-idmp001.COMPANYidm.org':
Directory Service: Unsecure port (389): OK
Directory Service: Secure port (636): OK
Kerberos KDC: TCP (88): OK
Kerberos Kpasswd: TCP (464): OK
HTTP Server: Unsecure port (80): OK
HTTP Server: Secure port (443): OK
The following list of ports use UDP protocol and would need to be
checked manually:
Kerberos KDC: UDP (88): SKIPPED
Kerberos Kpasswd: UDP (464): SKIPPED
Connection from replica to master is OK.
Start listening on required ports for remote master check
Listeners are started. Use CTRL+C to terminate the listening part after
the test.
Please run the following command on remote master:
/usr/sbin/ipa-replica-conncheck --replica
eu-idmp001.COMPANYidm.org