I'm now noticing in /var/log/dirsrv/slapd-*/errors a bunch of lines like this:

WARN - csngen_new_csn - Too much time skew (-15785961 secs). Current seqnum=1a22

And so on.  All 3 servers are correctly time-synced to our internal NTP server, so could this be something internal? A counter of some kind?


photo
Bret Wortman
Founder, Damascus Products, LLC

855-644-2783 | bret@wrapbuddies.co

http://wrapbuddies.co/

70 Main St. Suite 23 Warrenton, VA 20186
On Mar 26 2019, at 11:43 am, Bret Wortman via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:

On Mar 26 2019, at 11:10 am, Florence Blanc-Renaud <flo@redhat.com> wrote:
On 3/26/19 2:23 PM, Bret Wortman via FreeIPA-users wrote:
I broke out of it, but the two are still out of sync. Is there a way to
get past that?


photo
*Bret Wortman*
Founder, Damascus Products, LLC

855-644-2783 <tel:855-644-2783> | bret@wrapbuddies.co
<https://link.getmailspring.com/link/76FBB986-2615-4565-A74D-E3C1D7A38233@getmailspring.com/0?redirect=mailto%3Abret%40wrapbuddies.co&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

http://wrapbuddies.co/
<https://link.getmailspring.com/link/76FBB986-2615-4565-A74D-E3C1D7A38233@getmailspring.com/1?redirect=http%3A%2F%2Fwrapbuddies.co%2F&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

70 Main St. Suite 23 Warrenton, VA 20186

<https://link.getmailspring.com/link/76FBB986-2615-4565-A74D-E3C1D7A38233@getmailspring.com/2?redirect=http%3A%2F%2Ffacebook.com%2Fwrapbuddiesco&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>
<https://link.getmailspring.com/link/76FBB986-2615-4565-A74D-E3C1D7A38233@getmailspring.com/3?redirect=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fbretwortman&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn> <https://link.getmailspring.com/link/76FBB986-2615-4565-A74D-E3C1D7A38233@getmailspring.com/4?redirect=http%3A%2F%2Ftwitter.com%2Fwrapbuddiesco&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn> <https://link.getmailspring.com/link/76FBB986-2615-4565-A74D-E3C1D7A38233@getmailspring.com/5?redirect=http%3A%2F%2Finstagram.com%2Fwrapbuddies&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

On Mar 26 2019, at 9:07 am, Rob Crittenden <rcritten@redhat.com> wrote:

Bret Wortman via FreeIPA-users wrote:

Oops. I spoke too soon. The one I thought I fixed is now just
scrolling
"No status yet" over and over...


You can break out of that. There is a bug where we are checking the
wrong status. I can't find the BZ at the moment but IIRC it will be
fixed in the next release.


The BZ is https://bugzilla.redhat.com/show_bug.cgi?id=1666843

rob



photo
*Bret Wortman*
Founder, Damascus Products, LLC

855-644-2783 <tel:855-644-2783> | bret@wrapbuddies.co
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/0?redirect=mailto%3Abret%40wrapbuddies.co&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

http://wrapbuddies.co/
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/1?redirect=http%3A%2F%2Fwrapbuddies.co%2F&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

70 Main St. Suite 23 Warrenton, VA 20186

<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/2?redirect=http%3A%2F%2Ffacebook.com%2Fwrapbuddiesco&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/3?redirect=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fbretwortman&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/4?redirect=http%3A%2F%2Ftwitter.com%2Fwrapbuddiesco&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/5?redirect=http%3A%2F%2Finstagram.com%2Fwrapbuddies&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

On Mar 26 2019, at 8:54 am, Bret Wortman
<bret.wortman@damascusgrp.com>
wrote:

One had a clock skew error (fixed), but the other non-CA replica
shows:

ipa3.spx.net:
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/6?redirect=ipa3.spx.net%3A&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>replica
  last init status: None
  last init ended: 1970-01-01 00:00:00+00:00
  last update status: Error (3) Replication error acquiring
replica: Unable to acquire replica: permission denied. The bind dn
does not have permission to supply replication updates to the
replica. Will retry later. (permission denied)

Do I need to re-init this replica from scratch (as in, remove it,
unbind it from the servers, re-add it as a client and then
re-promote it)?

The "init" status is updated when a full reinitialization is done, not
during normal replication updates. The "last update status" is the
relevant information in your case.

Ours is still showing that status from 2019-03-13.


Can you check if each master has a valid keytab and is able to use this
keytab to authenticate to the other masters? See
https://www.freeipa.org/page/Troubleshooting/Directory_Server#Replication_issues

The two ldapsearches worked on both replicas having issues.


What is your 389-ds version?

1.3.8.4-22 on CentOS 7.

You may check that the group "cn=replication
managers,cn=sysaccounts,cn=etc,$BASEDN" contains as member all your
replication principals, for instance:

dn: cn=replication managers,cn=sysaccounts,cn=etc,$BASEDN
cn: replication managers
member:
krbprincipalname=ldap/master.domain.com@DOMAIN.COM,cn=services,cn=accounts,$BASEDN
member:
krbprincipalname=ldap/replica.domain.com@DOMAIN.COM,cn=services,cn=accounts,$BASEDN

and that the group is configured as nsds5replicabinddngroup in
cn=replica,cn=dc\3Ddomain\2Cdc\3Dcom,cn=mapping tree,cn=config

If you have an older version, I believe nsds5replicabinddn is used
instead of nsds5replicabinddngroup.


To try to get replication flowing again, I stopped and started IPA on the ipa5 server (using ipactl stop && ipactl start), and now:

# ipa-replica-manage list
ipa3.my.net: master
ipa4.my.net: master
ipa5.my.net: master
# ipa-replica-manage list -v ipa5.spx.net
#

In fact, ipa-replica-manage list with a hostname on any of our servers returns nothing now.

HTH,
flo


photo
*Bret Wortman*
Founder, Damascus Products, LLC

855-644-2783 <tel:855-644-2783> | bret@wrapbuddies.co
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/7?redirect=mailto%3Abret%40wrapbuddies.co&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

http://wrapbuddies.co/
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/8?redirect=http%3A%2F%2Fwrapbuddies.co%2F&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

70 Main St. Suite 23 Warrenton, VA 20186

<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/9?redirect=http%3A%2F%2Ffacebook.com%2Fwrapbuddiesco&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/10?redirect=http%3A%2F%2Fwww.linkedin.com%2Fin%2Fbretwortman&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/11?redirect=http%3A%2F%2Ftwitter.com%2Fwrapbuddiesco&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>
<https://link.getmailspring.com/link/1183D1DD-2462-44D7-A501-D9F2A79E8D1C@getmailspring.com/12?redirect=http%3A%2F%2Finstagram.com%2Fwrapbuddies&recipient=ZnJlZWlwYS11c2Vyc0BsaXN0cy5mZWRvcmFob3N0ZWQub3Jn>

On Mar 26 2019, at 8:47 am, Rob Crittenden <rcritten@redhat.com>
wrote:

Bret Wortman via FreeIPA-users wrote:

Looks like I've somehow managed to get my 3 IPA servers out
of sync:

[root@ipa3 ~]# ipa-replica-manage list
ipa3.my.net:master
ipa4.my.net:master
ipa5.my.net:master
[root@ipa3 ~]# ipa host-find solr14.my.net
---------------
0 hosts matched
---------------
----------------------------
Number of entries returned 0
----------------------------

On ipa4:
[root@ipa3 ~]# ipa host-find solr14.my.net
---------------
1 hosts matched
---------------
  Host name: solr14.my.net
----------------------------
Number of entries returned 1
----------------------------

On ipa5:
[root@ipa3 ~]# ipa host-find solr14.my.net
---------------
1 hosts matched
---------------
  Host name: solr14.my.net
  Principal name: host/solr14.my.net@MY.NET
<mailto:host/solr14.my.net@MY.NET>
  :
  :
----------------------------
Number of entries returned 1
----------------------------

So they've obviously stopped talking. What's the right way
to get them
back in sync and ensure that they don't drift again? Is there a
replication entry that's "stuck" and causing this?


On each master run: ipa-replica-manage list -v `hostname`

That will give you the replication status.

You can try to wake up an agreement with: ipa-replica-manage
force-sync
--from <host>

rob

Sent from Mailspring


_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org

Sent from Mailspring

_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Sent from Mailspring
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Sent from Mailspring