Hi Everyone,
I was wondering what experiences people have had with large groups (> 100k members) in 389 DS? Particularly interested in loading, managing and syncing them. WRT syncing – how do people efficiently sync large groups? Most sync utilities sync at the attribute level; if the changed attribute (eg member) is multivalued, it just replaces all values. That’s OK if there’s only a few values, but is not efficient when there are a large number of them. A more efficient way would be to diff the 2 attributes and add/delete the differences; does anyone know of any sync tools that do something like this?
Background: I have a few particularly large groups of > 500k members that are currently handled in a DBMS, but want to migrate them to LDAP instead. When I try to load them via ldapmodify, doing an add:member per member was going to take more than 24 hrs at rate of processing at the time of abort. Trying to add all members instead, with a single add:member and listing all members after that instruction, eventually ended with an Operations Error. Turning on housekeeping error level showed it was getting “Lock table is out of available lock entries” error – I’m in the process of retrying with adjusted nsslapd-db-locks in cn=config,cn=ldbm database,cn=plugins,cn=config.
Thanks, Trev
_________________________________________________ Trevor Fong Senior Programmer Analyst Information Technology | Engage. Envision. Enable. The University of British Columbia trevor.fong@ubc.camailto:trevor.fong@ubc.ca | 1-604-827-5247tel:604-827-5247 | it.ubc.cahttp://it.ubc.ca
On 04/26/2018 08:06 PM, Fong, Trevor wrote:
Hi Everyone,
I was wondering what experiences people have had with large groups (> 100k members) in 389 DS?
Particularly interested in loading, managing and syncing them.
WRT syncing – how do people efficiently sync large groups? Most sync utilities sync at the attribute level; if the changed attribute (eg member) is multivalued, it just replaces all values. That’s OK if there’s only a few values, but is not efficient when there are a large number of them. A more efficient way would be to diff the 2 attributes and add/delete the differences; does anyone know of any sync tools that do something like this?
389-ds does syncing/replicaten by replictaing the operation, so if you add one member only this modify/add operation will be replicated and applied. So the efficiency depends on your client application, if it always does a replace the the full group will be replicated, if it does add/del of single members the replication footprint is very small
Background:
I have a few particularly large groups of > 500k members that are currently handled in a DBMS, but want to migrate them to LDAP instead.
When I try to load them via ldapmodify, doing an add:member per member was going to take more than 24 hrs at rate of processing at the time of abort.
Trying to add all members instead, with a single add:member and listing all members after that instruction, eventually ended with an Operations Error. Turning on housekeeping error level showed it was getting “Lock table is out of available lock entries” error – I’m in the process of retrying with adjusted nsslapd-db-locks in cn=config,cn=ldbm database,cn=plugins,cn=config.
Thanks,
Trev
*Trevor Fong*
/Senior Programmer Analyst/
Information Technology | Engage. Envision. Enable.
*The University of British Columbia*
trevor.fong@ubc.ca mailto:trevor.fong@ubc.ca | 1-604-827-5247 tel:604-827-5247 | it.ubc.ca http://it.ubc.ca
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org
389-users@lists.fedoraproject.org
-
Fong, Trevor -
Ludwig Krispenz