On Jun 21, 2023, at 18:07, Rob Crittenden <rcritten@redhat.com> wrote:

Joe Rhodes via FreeIPA-users wrote:
Hello all!

I have a CentOS 7 based FreeIPA system that I’m migrating to Rocky 9.
 As suggested, I’ve created a Rocky 8 instance replica first.

As I’ve been working on this (in a dev environment first), I’ve gotten
myself into a state where I have two servers in the config that I cannot
delete.  (The VMs have been uninstalled and deleted.)

ipa server-find

---------------------

7 IPA servers matched

---------------------

  Server name: ia-ipa-1.dev.purestake.tech

  Min domain level: 0

  Max domain level: 1


  Server name: ia-ipa-2.dev.purestake.tech

  Min domain level: 0

  Max domain level: 1


  Server name: joe-rocky-8.dev.purestake.tech

  Min domain level: 1

  Max domain level: 1


  Server name: joe-rocky-9.dev.purestake.tech

  Min domain level: 1

  Max domain level: 1


  Server name: oh-ipa-1.dev.purestake.tech

  Min domain level: 0

  Max domain level: 1


  Server name: oh-ipa-2.dev.purestake.tech

  Min domain level: 0

  Max domain level: 1


  Server name: oh-ipa-21.dev.purestake.tech

  Min domain level: 1

  Max domain level: 1



The two servers I want to delete are  joe-rocky-9  and oh-ipa-21.

Trying to delete either give me:

ipa server-del joe-rocky-9.dev.purestake.tech

Removing joe-rocky-9.dev.purestake.tech from replication topology,
please wait...

ipa: ERROR: Server removal aborted: 


Replication topology in suffix 'domain' is disconnected:

Topology does not allow server ia-ipa-1.dev.purestake.tech to replicate
with servers:

    joe-rocky-9.dev.purestake.tech

Topology does not allow server ia-ipa-2.dev.purestake.tech to replicate
with servers:

    joe-rocky-9.dev.purestake.tech

Topology does not allow server joe-rocky-8.dev.purestake.tech to
replicate with servers:

    joe-rocky-9.dev.purestake.tech

Topology does not allow server joe-rocky-9.dev.purestake.tech to
replicate with servers:

    joe-rocky-8.dev.purestake.tech

    oh-ipa-1.dev.purestake.tech

    oh-ipa-2.dev.purestake.tech

    ia-ipa-1.dev.purestake.tech

    oh-ipa-21.dev.purestake.tech

    ia-ipa-2.dev.purestake.tech

Topology does not allow server oh-ipa-1.dev.purestake.tech to replicate
with servers:

    joe-rocky-9.dev.purestake.tech

Topology does not allow server oh-ipa-2.dev.purestake.tech to replicate
with servers:

    joe-rocky-9.dev.purestake.tech

Topology does not allow server oh-ipa-21.dev.purestake.tech to replicate
with servers:

    joe-rocky-9.dev.purestake.tech.


and attempting to delete, ignoring the replication topology:

ipa server-del joe-rocky-9.dev.purestake.tech --ignore-topology-disconnect

Removing joe-rocky-9.dev.purestake.tech from replication topology,
please wait...

ipa: ERROR: Not allowed on non-leaf entry


When I do a:  ipa topologysegment-find domain the server joe-rocky-9 is
not listed in any of the segments.

I believe the issue is I have a bunch of replication issues regarding
these two servers.  (I had been adding and removing them as I was
finding the right way to go about my upgrade)  This command shows both
of the servers:


ldapsearch "nsds5ReplConflict=*"


When I do the following search I see quite a few nsTombstone entries as
children, which I assume is what’s blocking me from removing this DN
(either using the ipa server-del command or the ldapdelete command).


ldapsearch -D "cn=Directory Manager” -W "(objectclass=nsTombstone)" dn



When I do this command:


ipa-replica-manage  list-ruv

Replica Update Vectors:

ia-ipa-1.dev.purestake.tech:389: 4

oh-ipa-1.dev.purestake.tech:389: 7

ia-ipa-2.dev.purestake.tech:389: 3

oh-ipa-2.dev.purestake.tech:389: 8

joe-rocky-8.dev.purestake.tech:389: 19

Certificate Server Replica Update Vectors:

ia-ipa-1.dev.purestake.tech:389: 6

joe-rocky-8.dev.purestake.tech:389: 20

ia-ipa-2.dev.purestake.tech:389: 5


I get the expected list of RUVs, without the two servers I want to
delete.  Only the serves that are really on-line and legit show up.  So
I cannot use the “clean-ruv” command because the bad servers don’t show
up with a replication ID.

When I do this:

ipa-replica-manage -p Extraordinary-northern-Conditioning-Idaho-7
clean-dangling-ruv


The server 'joe-rocky-9.dev.purestake.tech' appears to be offline.

The server 'oh-ipa-21.dev.purestake.tech' appears to be offline.

No dangling RUVs found



I see the two problematic entries timing out (as expected, since they
don’t exist).

I’m just not sure how to remove these two dead servers.  It seems like I
need to resolve or delete the nsTombstone children, but that doesn’t
seem to be possible.

I’m kind of wondering if I’m at a point where I’ll need to do an
ipa-backup/modify the ldif/ipa-restore to get rid of these?  I’m not
even sure that’s possible.

Any help would be greatly appreciated.  

I'd look at the topology in the UI to see visually how the servers are
connected. It sounds like the topology sees joe-rocky-9 as the only
thing connecting most of the replicas.

You can use ipa topology-segment to create new links to other servers
and that should allow removing it.

rob


Unfortunately, I don’t think that’s it.

The two servers I’m trying to remove are not required for the topology.  Both the GUI and the CLI show this:  

ipa topologysegment-find  domain

------------------

6 segments matched

------------------

  Segment name: ia-ipa-1.dev.purestake.tech-to-ia-ipa-2.dev.purestake.tech

  Left node: ia-ipa-1.dev.purestake.tech

  Right node: ia-ipa-2.dev.purestake.tech

  Connectivity: both


  Segment name: ia-ipa-1.dev.purestake.tech-to-oh-ipa-1.dev.purestake.tech

  Left node: ia-ipa-1.dev.purestake.tech

  Right node: oh-ipa-1.dev.purestake.tech

  Connectivity: both


  Segment name: ia-ipa-2.dev.purestake.tech-to-oh-ipa-2.dev.purestake.tech

  Left node: ia-ipa-2.dev.purestake.tech

  Right node: oh-ipa-2.dev.purestake.tech

  Connectivity: both


  Segment name: joe-rocky-8.dev.purestake.tech-to-oh-ipa-1.dev.purestake.tech

  Left node: joe-rocky-8.dev.purestake.tech

  Right node: oh-ipa-1.dev.purestake.tech

  Connectivity: both


  Segment name: joe-rocky-8.dev.purestake.tech-to-oh-ipa-21.dev.purestake.tech

  Left node: joe-rocky-8.dev.purestake.tech

  Right node: oh-ipa-21.dev.purestake.tech

  Connectivity: both


  Segment name: oh-ipa-1.dev.purestake.tech-to-oh-ipa-2.dev.purestake.tech

  Left node: oh-ipa-1.dev.purestake.tech

  Right node: oh-ipa-2.dev.purestake.tech

  Connectivity: both

----------------------------

Number of entries returned 6

----------------------------


joe-rocky-9 isn’t listed in the CLI output.  In the GUI, it’s just a green dot floating off by itself.  oh-ipa-21 has one domain connection to joe-rocky-8.  But I cannot delete that topology segment, probably for whatever reason is keeping me from deleting the server itself.