Hi,

excellent ! We are almost there. Thanks to you for the hard work.

We should not have two segments with connectivity "both"  for CA as you have already mentionned.

We should fix this just with the command line. If you delete the segment using:

ipa topologysegment-del ca <one of the segments>

this should work fine. If not, with my procedure it should work as well. Are you sure you are restarting the services after disabling the plugin ?

Thanks a lot.

German.



On Thu, Feb 14, 2019 at 11:36 AM dbischof--- via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
Hi German,

On Tue, 12 Feb 2019, German Parente via FreeIPA-users wrote:

> well, there's still a possibility to remove it manually. it's rather
> easy.
>
> ldapsearch -D "cn=directory manager" -W -b
> "cn=topology,cn=ipa,cn=etc,dc=example,dc=com"
>
> that will show all the entries in the topology subtree. You will find
> the one with "left-right" or "right-left" connectivity.
>
> Before deleting it, do this:
>
> ldapmodify -D "cn=directory manager" -W << EOF
> dn: cn=IPA Topology Configuration,cn=plugins,cn=config
> changetype: modify
> replace: nsslapd-pluginEnabled
> nsslapd-pluginEnabled: off
> EOF
>
> restart services.
>
> Delete the wrong entry with "ldapdelete" command.
>
> then, do this:
> ldapmodify -D "cn=directory manager" -W << EOF
> dn: cn=IPA Topology Configuration,cn=plugins,cn=config
> changetype: modify
> replace: nsslapd-pluginEnabled
> nsslapd-pluginEnabled: on
> EOF
>
> restart services.
>
> Check your segments again.
>
> if you have a subscription, please open a support case, ask for my help
> and I will fix that in your machines in a remote session.

thanks again for your hints. I guess, i'm out of the woods now:

---
$ ipa topologysegment-find domain
-----------------
1 segment matched
-----------------
   Segment name: ipa2.example.com-to-ipa1.example.com
   Left node: ipa2.example.com
   Right node: ipa1.example.com
   Connectivity: both
----------------------------
Number of entries returned 1
----------------------------

$ ipa topologysegment-find ca
------------------
2 segments matched
------------------
   Segment name: ipa2.example.com-to-ipa1.example.com
   Left node: ipa2.example.com
   Right node: ipa1.example.com
   Connectivity: both

   Segment name: ipa1.example.com-to-ipa2.example.com
   Left node: ipa1.example.com
   Right node: ipa2.example.com
   Connectivity: both
----------------------------
Number of entries returned 2
----------------------------

$ ipa-replica-manage -v list ipa1.example.com
ipa2.example.com: replica
   last init status: Error (0) Total update succeeded
   last init ended: 2019-01-11 20:02:40+00:00
   last update status: Error (0) Replica acquired successfully: Incremental update succeeded
   last update ended: 2019-02-14 09:05:01+00:00

$ ipa-replica-manage -v list ipa2.example.com
ipa1.example.com: replica
   last init status: None
   last init ended: 1970-01-01 00:00:00+00:00
   last update status: Error (0) Replica acquired successfully: Incremental update succeeded
   last update ended: 2019-02-14 09:05:00+00:00

$ ipa-csreplica-manage -v list ipa1.example.com
ipa2.example.com
   last init status: Error (0) Total update succeeded
   last init ended: 2019-02-13 13:43:48+00:00
   last update status: Error (0) Replica acquired successfully: Incremental update succeeded
   last update ended: 2019-02-14 09:02:30+00:00

$ ipa-csreplica-manage -v list ipa2.example.com
ipa1.example.com
   last init status: None
   last init ended: 1970-01-01 00:00:00+00:00
   last update status: Error (0) Replica acquired successfully: Incremental update succeeded
   last update ended: 2019-02-14 08:57:30+00:00
---

Identical output on both ipa1 and ipa2. In addition to your advice, I had
to re-create segment ipa1.example.com-to-ipa2.example.com using
topologysegment-add and I also did a

---
$ ipa-csreplica-manage re-initialize --from ipa1.example.com
---

on ipa2.

I tried to remove the ipa2.example.com-to-ipa1.example.com ca segment
using your recipe, but that breaks things, I had to re-create it
afterwards. Error messages in dirsrv logs gone.

It remains unclear to me, why removing the 2nd ca segment didn't work - so
just to make sure: One "Connectivity: both"-segment for each domain and ca
is sufficient (for my two-master-only-topology), right? Having 2 ca
segments won't hurt?

Once again: Your help was invaluable.

> On Tue, Feb 12, 2019 at 9:17 AM dbischof--- via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
>
>> On Mon, 11 Feb 2019, German Parente via FreeIPA-users wrote:
>>
>>> in fact, there's no sense to have "two segments" one from ipa1 <--> ipa2
>>> and other ipa1 --> ipa2.
>>>
>>> you should delete the segment that is showing "right-left" connectivity.
>>
>> that doesn't work, i tried that already:
>>
>> ---
>> $ ipa topologysegment-del ca ipa1.example.com-to-ipa2.example.com
>> ipa: ERROR: Server is unwilling to perform: Removal of Segment disconnects
>> topology. Deletion not allowed.
>> ---
>>
>> Tried on both masters.
>>
>>> On Mon, Feb 11, 2019 at 1:47 PM dbischof--- via FreeIPA-users <
>>> freeipa-users@lists.fedorahosted.org> wrote:
>>>
>>>> On Mon, 11 Feb 2019, German Parente via FreeIPA-users wrote:
>>>>
>>>>> don't forget "-r" to export. If not, replication metadata will not be
>>>>> exported and after the import, the replicas will not be in sync.
>>>>
>>>> thank you for your hints.
>>>>
>>>> Unfortunately, the replication/topology problem remains unsolved.
>>>>
>>>> Here's what i did:
>>>>
>>>> --- ipa1 (IPA running)
>>>> db2ldif.pl -Z EXAMPLE-COM -D "cn=Directory Manager" -r -w - -n ipaca -a
>>>> /tmp/foo.dif
>>>> ---
>>>>
>>>> Copied the file over to ipa2, then
>>>>
>>>> --- ipa2 (IPA not running)
>>>> ldif2db -Z EXAMPLE-COM -n ipaca -i foo.dif
>>>> ---
>>>>
>>>> Started IPA on ipa2, but still
>>>>
>>>> ---
>>>> $ ipa topologysegment-find ca
>>>> ------------------
>>>> 2 segments matched
>>>> ------------------
>>>>    Segment name: ipa2.example.com-to-ipa1.example.com
>>>>    Left node: ipa2.example.com
>>>>    Right node: ipa1.example.com
>>>>    Connectivity: both
>>>>
>>>>    Segment name: ipa1.example.com-to-ipa2.example.com
>>>>    Left node: ipa1.example.com
>>>>    Right node: ipa2.example.com
>>>>    Connectivity: left-right
>>>> ----------------------------
>>>> Number of entries returned 2
>>>> ----------------------------
>>>>
>>>> In case there's nothing obvious and easy left to be tried out, I'd
>>>> consider to uninstall IPA on ipa2, reinstall as client and promote
>>>> ipa2 to master again as described in the docs.
>>>>
>>>>> On Thu, Feb 7, 2019 at 3:46 PM dbischof--- via FreeIPA-users <
>>>>> freeipa-users@lists.fedorahosted.org> wrote:
>>>>>
>>>>>> On Wed, 6 Feb 2019, German Parente via FreeIPA-users wrote:
>>>>>>
>>>>>>> this is a bug in the product that might have been fixed already:
>>>>>>>
>>>>>>> Connectivity: left-right
>>>>>>>
>>>>>>> we cannot have these sort of connectivity.
>>>>>>>
>>>>>>> In ipa02 there's no replication agreement to ipa01 (for ipaca
>>>>>>> database).
>>>>>>>
>>>>>>> But as in ipa01 we see that the topology is showing "both" in the
>>>>>>> connectivity, I suggest to do export-import "off line" of the
>>>>>>> database. Then the topology subtree will be set in ipa02, exactly as
>>>>>>> in ipa01, and the topology plugin will create automatically the
>>>>>>> replication agreement that is missing now.
>>>>>>>
>>>>>>> export from ipa01 the backend ipaca and re-import it in ipa02. Then,
>>>>>>> start the server and check if now it's showing "both" in connectivity
>>>>>>> at ipa02 side.
>>>>>>
>>>>>> thank you for your hints.
>>>>>>
>>>>>> Unfortunately, I never did something like this before (and I can't
>>>>>> access the article you cited below). According to the Directory
>>>>>> Manager docs, it's probably something like
>>>>>>
>>>>>> ---
>>>>>> db2ldif.pl -Z EXAMPLE-COM -D "cn=Directory Manager" -w - -n ipaca -a /tmp/foo.dif
>>>>>> ---
>>>>>>
>>>>>> to export on running ipa1 and
>>>>>>
>>>>>> ---
>>>>>> ldif2db -Z EXAMPLE-COM -n ipaca -i /tmp/foo.dif
>>>>>> ---
>>>>>>
>>>>>> to import on ipa2 with IPA not running, right? Something else to be
>>>>>> taken into account to not break something (these are production
>>>>>> servers - my group is small but vigorous ;-)
>>>>>>
>>>>>>> On Wed, Feb 6, 2019 at 4:57 PM dbischof--- via FreeIPA-users
>>>>>>> <freeipa-users@lists.fedorahosted.org> wrote:
>>>>>>>
>>>>>>>> On Wed, 6 Feb 2019, German Parente via FreeIPA-users wrote:
>>>>>>>>
>>>>>>>>> have you tried to use "ipa-csreplica-manage re-initialize --from
>>>>>>>>> <replica1>" in replica1 ?
>>>>>>>>
>>>>>>>> Thanks for your answer.
>>>>>>>>
>>>>>>>> I already tried (on ipa2)
>>>>>>>>
>>>>>>>> ---
>>>>>>>> $ ipa-csreplica-manage re-initialize --from ipa1.example.com
>>>>>>>> ---
>>>>>>>>
>>>>>>>> which failed.
>>>>>>>>
>>>>>>>> Interestingly enough, the error message is
>>>>>>>>
>>>>>>>> ---
>>>>>>>> unexpected error: Replication agreement for ipa1.example.com
>>>>>>>> not found
>>>>>>>> ---
>>>>>>>>
>>>>>>>> And indeed:
>>>>>>>>
>>>>>>>> ---
>>>>>>>> $ ipa topologysegment-find ca
>>>>>>>> ------------------
>>>>>>>> 2 segments matched
>>>>>>>> ------------------
>>>>>>>>    Segment name: ipa2.example.com-to-ipa1.example.com
>>>>>>>>    Left node: ipa2.example.com
>>>>>>>>    Right node: ipa1.example.com
>>>>>>>>    Connectivity: both
>>>>>>>>
>>>>>>>>    Segment name: ipa1.example.com-to-ipa2.example.com
>>>>>>>>    Left node: ipa1.example.com
>>>>>>>>    Right node: ipa2.example.com
>>>>>>>>    Connectivity: left-right
>>>>>>>> ----------------------------
>>>>>>>> Number of entries returned 2
>>>>>>>> ----------------------------
>>>>>>>> ---
>>>>>>>>
>>>>>>>> The Web UI topology graph doesn't reflect this, btw.
>>>>>>>>
>>>>>>>> Isn't the 2nd segment obsolete and probably causing my CS
>>>>>>>> replication issues? Just remove it?
>>>>>>>>
>>>>>>>>> You could also re-init off line by using this article:
>>>>>>>>>
>>>>>>>>> https://access.redhat.com/solutions/140483
>>>>>>>>>
>>>>>>>>> only for ipaca backend.
>>>>>>>>>
>>>>>>>>> On Wed, Feb 6, 2019 at 11:31 AM dbischof--- via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
>>>>>>>>>
>>>>>>>>>> On Wed, 6 Feb 2019, dbischof--- via FreeIPA-users wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, 6 Feb 2019, Florence Blanc-Renaud via FreeIPA-users wrote:
>>>>>>>>>>>
>>>>>>>>>>>>  On 2/5/19 4:17 PM, dbischof--- via FreeIPA-users wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>   my IPA system consists of 2 masters (ipa1 and ipa2, both on
>>>>>>>>>>>>>   FreeIPA 4.6.4) with their own self-signed CAs, one of them
>>>>>>>>>>>>>   being the certificate renewal master (ipa1). The system has
>>>>>>>>>>>>>   been running for years and has been migrated from an IPA 3
>>>>>>>>>>>>>   system. Both IPA servers are on domain level 1.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Problem: CS replication failed, probably months ago.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   --- ipa1 ---
>>>>>>>>>>>>>   $ ipa-csreplica-manage -v list ipa1.example.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>   ipa2.example.com
>>>>>>>>>>>>>      last init status: None
>>>>>>>>>>>>>      last init ended: 1970-01-01 00:00:00+00:00
>>>>>>>>>>>>>      last update status: Error (-1) Problem connecting to replica - LDAP error: Can't contact LDAP server (connection error)
>>>>>>>>>>>>>      last update ended: 1970-01-01 00:00:00+00:00
>>>>>>>>>>>>>
>>>>>>>>>>>>>   --
>>>>>>>>>>>>>   $ ipa-csreplica-manage -v list ipa2.example.com
>>>>>>>>>>>>>
>>>>>>>>>>>>>   [no output]
>>>>>>>>>>>>>   ----
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Same on ipa2.
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Probably related:
>>>>>>>>>>>>>
>>>>>>>>>>>>>   ---
>>>>>>>>>>>>>   ERR - slapi_ldap_bind - Error: could not send startTLS
>>>>>>>>>>>>>   request: error -1 (Can't contact LDAP server) errno 107
>>>>>>>>>>>>>   (Transport endpoint is not connected)
>>>>>>>>>>>>>   ---
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Every 5 mins in /var/log/dirsrv/slapd-EXAMPLE-COM/errors.
>>>>>>>>>>>>>   However, these error messages could refer to
>>>>>>>>>>>>>   ipa3.example.com, a master i deleted long (> 2 years) ago:
>>>>>>>>>>>>>
>>>>>>>>>>>>>   ---
>>>>>>>>>>>>>   $ ipa-replica-manage list-ruv
>>>>>>>>>>>>>
>>>>>>>>>>>>>   Replica Update Vectors:
>>>>>>>>>>>>>            ipa2.example.com:389: 10
>>>>>>>>>>>>>            ipa1.example.com:389: 9
>>>>>>>>>>>>>   Certificate Server Replica Update Vectors:
>>>>>>>>>>>>>            ipa2.example.com:389: 11
>>>>>>>>>>>>>            ipa1.example.com:389: 91
>>>>>>>>>>>>>            ipa2.example.com:7389: 96
>>>>>>>>>>>>>            ipa3.example.com:7389: 97
>>>>>>>>>>>>>   ---
>>>>>>>>>>>>>
>>>>>>>>>>>>>   How do i track this down and resolve the problem?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>  please find more information re. 389-ds troubleshooting:
>>>>>>>>>>>>  https://www.freeipa.org/page/Troubleshooting/Directory_Server
>>>>>>>>>>>
>>>>>>>>>>> I checked for the common problems described in that page already,
>>>>>>>>>>> but to no avail. I did, however, successfully manage to remove
>>>>>>>>>>> replication references to ipa3 using "ipa-replica-manage
>>>>>>>>>>> clean-dangling-ruv":
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> $ ipa-replica-manage list-ruv
>>>>>>>>>>> Replica Update Vectors:
>>>>>>>>>>>         ipa1.example.com:389: 9
>>>>>>>>>>>         ipa2.example.com:389: 10
>>>>>>>>>>> Certificate Server Replica Update Vectors:
>>>>>>>>>>>         ipa1.example.com:389: 91
>>>>>>>>>>>         ipa2.example.com:389: 11
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> The error message
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> [06/Feb/2019:10:38:52.095489260 +0100] - ERR - slapi_ldap_bind -
>>>>>>>>>>> Error: could not send startTLS request: error -1 (Can't contact LDAP
>>>>>>>>>>> server) errno 107 (Transport endpoint is not connected)
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> on ipa1 is still in the logs. Additionally, while cleaning ruvs:
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> [06/Feb/2019:10:32:31.029394375 +0100] - ERR - NSMMReplicationPlugin
>>>>>>>>>>> - bind_and_check_pwp -
>>>>>>>>>>> agmt="cn=cloneAgreement1-ipa1.example.com-pki-tomcat" (ipa2:7389) -
>>>>>>>>>>> Replication bind with SIMPLE auth failed: LDAP error -1 (Can't
>>>>>>>>>>> contact LDAP server) ()
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> The ldapsearch queries described in the above page can be carried
>>>>>>>>>>> out successfully on both servers:
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>> [...]
>>>>>>>>>>> # search result
>>>>>>>>>>> search: 4
>>>>>>>>>>> result: 0 Success
>>>>>>>>>>>
>>>>>>>>>>> #  numResponses: 2
>>>>>>>>>>> #  numEntries: 1
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> Also, no DNS issues, wrong entries /etc/hosts, time
>>>>>>>>>>> differences or log messages related to SASL issues.
>>>>>>>>>>>
>>>>>>>>>>> Maybe a wrong key or certificate somewhere?
>>>>>>>>>>
>>>>>>>>>> update: ipa-checkcerts.py shows
>>>>>>>>>>
>>>>>>>>>> ---
>>>>>>>>>> [...]
>>>>>>>>>> Failures:
>>>>>>>>>> ipa: INFO: Unable to find request for serial 268304391
>>>>>>>>>> Unable to find request for serial 268304391
>>>>>>>>>> ipa: INFO: Unable to find request for serial 268304394
>>>>>>>>>> Unable to find request for serial 268304394
>>>>>>>>>> ipa: INFO: Unable to find request for serial 268304393
>>>>>>>>>> Unable to find request for serial 268304393
>>>>>>>>>> ipa: INFO: Unable to find request for serial 268304392
>>>>>>>>>> Unable to find request for serial 268304392
>>>>>>>>>> ipa: INFO: Subject O=EXAMPLE.COM,CN=ipa2.example.com and template
>>>>>>>>>> subject CN=ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>>>>>>>>> Subject O=EXAMPLE.COM,CN=ipa2.example.com and template subject CN=
>>>>>>>>>> ipa1.example.com,O=EXAMPLE.COM do not match for serial 57
>>>>>>>>>> ---
>>>>>>>>>>
>>>>>>>>>> So there is a certificate issue.


Mit freundlichen Gruessen/With best regards,

--Daniel.
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org