389-devel May 2016

389-devel@lists.fedoraproject.org

8 participants
58 discussions

48820 and 48853: Fixed links
by William Brown 26 May '16

26 May '16

Sorry, I realised I had a file in these that was in the wrong patch. Fixed patches here: https://fedorahosted.org/389/attachment/ticket/48820/0002-Ticket-48820-Proo… https://fedorahosted.org/389/attachment/ticket/48853/0003-Ticket-48853-Prer… -- Sincerely, William Brown Software Engineer Red Hat, Brisbane

1 0

Please review: 48853, experimental ds-setup(.py)
by William Brown 25 May '16

25 May '16

https://fedorahosted.org/389/ticket/48853 https://fedorahosted.org/389/attachment/ticket/48853/0003-Ticket-Prerelease… https://fedorahosted.org/389/attachment/ticket/48853/0001-Ticket-Pre-releas… -- Sincerely, William Brown Software Engineer Red Hat, Brisbane

1 0

Please review: 48820 new api style for lib389
by William Brown 25 May '16

25 May '16

Hi, This is split into two parts. https://fedorahosted.org/389/ticket/48820 First, is a set of python 3 fixes: https://fedorahosted.org/389/attachment/ticket/48820/0001-Ticket-Fixes-for-… Second is the new lib389 api I would like us to move towards. https://fedorahosted.org/389/attachment/ticket/48820/0002-Ticket-48820-Proo… This also cleans up and moves some other cli tools around (But I can break that out if we want). I included it because I adapted some of the tools to use the new style. I the best examples are in backend.py, config.py. The parent classes come from _mapped_object.py -- Why would you write this? Lets look at lib389 right now. We have an api that is very inconsistent. Some have get/set, some getProperties/setProperties. Some take a single value, some take a key, some take an array. Some are based on passing an identifer the function to act on, some are just singletons. Each one is attached to DirSrv, and makes a monolithic and huge api. In the end, it looks like lib389 is annoying to work with, and our own team has resorted to calling inst.add_s/modify_s on values even under cn=config: and we have a inst.config.get/set already! -- What does the new api look like? The idea is that all of our configurations in cn=config derive two styles: First, a static once off object, with a known basedn and properties. (ie cn=config) The second is a subtree of objects each with the same attribute types, but that implement many instances. (IE backends) This new way of handling this manages both types. The idea is to share and re-use as much as possible. Lets have a look at config.py. This is now a subclass of DSLdapObject. class Config(DSLdapObject): .... def __init__(self, conn, batch=False): super(Config, self).__init__(instance=conn, batch=batch) self._dn = DN_CONFIG Our super type, defines these methods: def __unicode__(self): def __str__(self): def set(self, key, value): def get(self, key): def remove(self, key): def delete(self): So we can now, just by deriving the type get access to string printing: config = Config(instance) print(config) >>> 'cn=config' We get a set / get on keys config.set('nsslapd-accesslog-level', '1') r = config.get('nsslapd-rootdn') Remove will remove a value for the attr, and delete will delete the object (unless you set self._protected, then it cannot be deleted. This is the default) A future idea is the batch flag. This will make it so that: config.set('k', 'v') config.set('x', 'y') config.commit() <<-- This actually does the ldap mod of k, x at once. This lets us make lots of changes in a more efficient way, and some values need to be updated in sync. And all we had to do was override self._basedn in class Config()! and we picked up so many functions straight out. So lets have a look at the backends now. The backends (note the plural) is derived from DSLdapObjects class Backends(DSLdapObjects): def __init__(self, instance, batch=False): super(Backends, self).__init__(instance=instance, batch=False) self._objectclasses = [BACKEND_OBJECTCLASS_VALUE] self._create_objectclasses = self._objectclasses + ['top', 'extensibleObject' ] self._filterattrs = ['cn', 'nsslapd-suffix', 'nsslapd-directory'] self._basedn = DN_LDBM self._childobject = Backend self._rdn_attribute = 'cn' self._must_attributes = ['nsslapd-suffix', 'cn'] Here we define that our "Backends" all assert certain properties. * They can be found with a certain objectclass * they must be created with a set of classes. * We can uniquely identify them based on the filterattributes. * They are all found under some basedn * They are named off the "cn" attribute * They must contain a cn and a nsslapd-suffix The _childobject type defines what the *single* backend instance is. Because of the inheritence, Backends already gains: def list(self): def get(self, selector): def create(self, rdn=None, properties=None): Additionally, there is an internal method (that will be explained below) def _validate(self, rdn, properties): Just from setting the attributes of the class, we can now list all backends on the system: bes = Backends(inst) print(bes.list()) We can select a backend based on one of the values of a matching attribute in _filterattrs be = bes.get('userRoot') be = bes.get('dc=example,dc=com') be = bes.get('/var/lib/dirsrv/slapd-localhost/db/userRoot') The be instance we get back in the Backend type. This is derive from DSLdapObject: Just like our config. This means it has all the same methods, such as get, set, __unicode__, as our config! We can see how little code it takes to do this: class Backend(DSLdapObject): def __init__(self, instance, dn=None, batch=False): super(Backend, self).__init__(instance, dn, batch) self._naming_attr = 'cn' That is the *entire* definition of Backend. Finally, the true power of DSLdapObjects is when we go to *create* a new instance. Creation of objects is something that in lib389 is hard. We do a lot of validation and checking to be sure of some things. Because we are deriving this type, we can already do a baseline of validation in our creation. The pattern in: bes = Backends(inst) be = bes.create(properties={'nsslapd-suffix': suffix, 'cn': 'userRoot'}) That's it. The reason for the _validate method on Backends is it allows us to hook and do custom validation for the type. In this case, backends can also take the lib389 properties style dictionary, and _validate will re-map the attributes correctly: be = bes.create(properties={'suffix': suffix, 'name': 'userRoot'}) _validate will help us by checking: * Is properties a valid dictionary of types? * Do we have a valid rdn (from self._rdn_attribute) * Do we have all the values of self._must_attributes satisfied? * Is our rdn going to be utf-8 (which python 3 expects?) At first look it seems like it could be a complex api. But you consider the needed work to extend and create say: class RSAEncrption(DSLdapObject): self.__init__(self, instance=None, batch=False): super(RSAEncryption, self).__init__(instance, batch) self._basedn = 'cn=RSA,cn=encryption,cn=config' And that's it. No more having to write modify, add, etc. We can easily, quickly, and confidently map our Directory Server configuration types into lib389. In the future with rest389 this will pay itself off massively, as a consistent, clean, reliable api is going to make creation and deployment of a rest admin console much, much more effecient. -- Isn't this going to end up nearly being a complete rewrite. Yes. But it needs to happen. Lib389 is straining, and hard to edit right now. We should improve this. -- But aren't there risks here of breaking all our tests? Yes. But there is a solution. In backend.py youll note I have: class BackendLegacy(object): This is the *original* Backend type that DirSrv attaches too. It's still accesible: def __add_brookers__(self): ... from lib389.backend import BackendLegacy as Backend This way, we can rename our existing types to <NAME>Legacy, and still use them in tests. I will add a "deprecation" flag to them, and if we decide to accept the new style of api I will begin to not only re-write our existing types, but our tests that rely on them. This way we can stage the transition over time. -- Are there any other wins here? Yes. I basically finished the port of lib389 to python3 in the process of this. -- Does that mean we can merge this without breaking our existing code and tests? Yes it does! -- Sincerely, William Brown Software Engineer Red Hat, Brisbane

1 0

Please review: #48275 - search returns no entry when OR filter component contains non readable attribute
by Ludwig Krispenz 25 May '16

25 May '16

https://fedorahosted.org/389/ticket/48275 https://fedorahosted.org/389/attachment/ticket/48275/0001-Ticket-48275-corr… https://fedorahosted.org/389/attachment/ticket/48275/0002-testcase-for-tick… The fix passes the ported TET filter test and the ticket specific test, please take your time to review and comment, I'll be out a few days and answer whenI'm back -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill

1 0

Re: Please review: ticket #48766 Replication changelog can incorrectly skip over updates
by Ludwig Krispenz 25 May '16

25 May '16

On 05/24/2016 07:00 PM, thierry bordaz wrote: > > > On 05/24/2016 05:24 PM, Ludwig Krispenz wrote: >> >> On 05/24/2016 04:20 PM, thierry bordaz wrote: >>> Hi Ludwig, >>> >>> Thanks for your explanation. The design looks very good. I think it >>> would be good to put into the code (especially >>> clcache_adjust_anchorcsn) the reference to the related design paragraph. >>> >>> There is something I do not understand in clcache_skip_change. >>> My understanding is that this is the only place where the >>> consumer_maxcsn is updated. >>> But there are several conditions, if we decide to skip the update >>> that the consumer_maxcsn is not updated. >>> One of them is 'rid == buf->buf_consumer_rid'. >>> Does that mean that the consumer_maxcsn remains unchanged for the >>> RID of the consumer ? >> >> the condition is: >> if( rid == buf->buf_consumer_rid && buf->buf_ignoreConsumerRID) >> >> so it will only be skipped if we determined that we don't need to >> send anything for the consumers own rid > > Ok. I get it. thanks. > > > An other question regarding the update of buf_consumer_ruv. > My understanding is that it contains the initial consumer RUV. > But later it is used in > clcache_refresh_consumer_maxcsns/clcache_refresh_local_maxcsn to fill > consumer_maxcsn. > > consumer_maxcsn are updated with the non skipped updates to reflect > the current status of the consumer. > But when consumer_maxcsns/clcache_refresh_local_maxcsn are called my > understanding is that consumer_maxcsn are reset with buf_consumer_ruv > (initial consumer RUV). > Do I get it right ? no :-) At least I think I have implemented it differently. The consumer_maxcsn is set in /clcache_refresh_consumer_maxcsns(), which is only called at the first buffer load, not at the relaod. In //clcache_refresh_local_maxcsns it is only set if we have to add a RID to the cscb list, but then no change for this rid was sent before. / > > thanks > thierry >> >>> >>> thanks >>> thierry >>> >>> >>> >>> >>> On 05/24/2016 09:22 AM, Ludwig Krispenz wrote: >>>> Hi, >>>> >>>> On 05/23/2016 06:29 PM, thierry bordaz wrote: >>>>> >>>>> >>>>> On 05/23/2016 03:06 PM, Ludwig Krispenz wrote: >>>>>> This is the latest version of the "changelog buffer processing" >>>>>> fixes. >>>>>> >>>>>> >>>>>> https://fedorahosted.org/389/ticket/48766 >>>>>> >>>>>> https://fedorahosted.org/389/attachment/ticket/48766/0001-reworked-clcach-b… >>>>>> >>>>>> >>>>>> The background for the fix is here, I would like to get feedback >>>>>> on this as well to clarify what is unclear >>>>>> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-state… >>>>>> >>>>>> >>>>> >>>>> Hello Ludwig, >>>>> >>>>> I have not yet reviewed the patch. I was looking at the design. >>>>> >>>>> Regarding your note: >>>>> http://www.port389.org/docs/389ds/design/changelog-processing-in-repl-state…. >>>> If you refer to this part: >>>> >>>> >>>> Special case: RID of the consumer in the current >>>> replication session >>>> >>>> If the consumer in the replication session is also a master its RID >>>> will be contained at least in the consumerRUV. If it is also in the >>>> supplier RUV the question is if it should be considered in the >>>> decision if updates should be sent. Normally a master has the >>>> latest changes applied to itself, so there would be no need to >>>> check and send updates for its RID. But there can be scenarios >>>> where this is not the case: if the consumer has been restored from >>>> an older backup the latest csn for its own RID might be older than >>>> changes available on other servers. >>>> >>>> |NOTE: The current implementation ignores anchorCSNs based on the consumer RID. If, by chance, the anchor csn used is older than this csn, the changes will be sent, but they also ca nbe lost. >>>> | >>>> >>>> this referres to the "current" implementation before the fix, the >>>> doc started as a post-design doc, and it shoul dbe correctedd >>>> with the fix the if the supplier has newer changes for the >>>> consumerRID than the consumer it will be reflected in the anchor >>>> csn calculation. >>>> >>>> >>>>> >>>>> It is said that the anchorCSN will not be the from the >>>>> consumerRID. What is the mechanism that guaranty that the consumer >>>>> will receive all the updates it was the originator ? >>>>> >>>>> thanks >>>>> thierry >>>>> -- >>>>> 389-devel mailing list >>>>> 389-devel(a)lists.fedoraproject.org >>>>> http://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org >>>>> >>>> >>>> -- >>>> Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn, >>>> Commercial register: Amtsgericht Muenchen, HRB 153243, >>>> Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill >>>> >>>> >>>> -- >>>> 389-devel mailing list >>>> 389-devel(a)lists.fedoraproject.org >>>> http://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org >>> >>> >>> >>> -- >>> 389-devel mailing list >>> 389-devel(a)lists.fedoraproject.org >>> http://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org >> >> -- >> Red Hat GmbH,http://www.de.redhat.com/, Registered seat: Grasbrunn, >> Commercial register: Amtsgericht Muenchen, HRB 153243, >> Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill >> >> >> -- >> 389-devel mailing list >> 389-devel(a)lists.fedoraproject.org >> http://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org > > > > -- > 389-devel mailing list > 389-devel(a)lists.fedoraproject.org > http://lists.fedoraproject.org/admin/lists/389-devel@lists.fedoraproject.org -- Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill

2 2

Initial review: ticket #48069 - CI test - filter
by Ludwig Krispenz 24 May '16

24 May '16

Hi, here is my current state of porting the TET filter suite, https://fedorahosted.org/389/ticket/48069 After getting there I think it can probably be simplified, but before just going to change things I like to get your feedback and suggestions for improvements Some issues I have: - in the original TET suite there are tons of files data, messages, errors .... I tried to integrate these into the tests and not rely on toom many files, but there are stilll two ldif fles in the data directory - what is best practice ? - the MR filter tests in filter.sh and vfilter.sh are forced to PASS. I omitted them and have created a separate ticket: https://fedorahosted.org/389/ticket/48851 https://fedorahosted.org/389/attachment/ticket/48069/0001-add-costants-for-… https://fedorahosted.org/389/attachment/ticket/48069/0001-add-test-data-for… https://fedorahosted.org/389/attachment/ticket/48069/0001-initial-port-of-T… https://fedorahosted.org/389/attachment/ticket/48069/0001-add-test-data-for… Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Paul Argiry, Charles Cachera, Michael Cunningham, Michael O'Neill

1 0

Please review 48745: Matching Rule caseExactIA5Match indexes incorrectly values with upper cases
by thierry bordaz 23 May '16

23 May '16

Hello, The patch for this ticket is in plugin_mr.c. It also contains minor changes in CI tests that failed to start or was unstable. thanks thierry ticket: https://fedorahosted.org/389/ticket/48745 patch: https://fedorahosted.org/389/attachment/ticket/48745/0002-Ticket-48745-Matc… test case: https://fedorahosted.org/389/attachment/ticket/48745/ticket48745_test.py

1 0

Please review 48844: Regression introduced in matching rules by DS 48746
by thierry bordaz 23 May '16

23 May '16

Hello, The patch for the ticket https://fedorahosted.org/389/ticket/48746 <https://fedorahosted.org/389/ticket/48745>, introduced a regression. This patch contained two parts: 1. Hardening of MR code to prevent a crash when index_create function do not set SLAPI_PLUGIN_OBJECT 2. rewriting of the code looking for MR plugin that can handle a given OID The Part 2 of the code was wrong because it did not take into consideration old and new MR plugin style that register their indexer create function in a different way. The Patch for 48844 basically revert the second part of the patch 48746. thanks thierry ticket: https://fedorahosted.org/389/ticket/48844 fix: https://fedorahosted.org/389/attachment/ticket/48844/0001-Ticket-48844-Regr… test case: https://fedorahosted.org/389/attachment/ticket/48844/ticket48844_test.py

1 0

Please review: 48850 fix memory leaks in pwdhash and ns-slapd
by William Brown 23 May '16

23 May '16

https://fedorahosted.org/389/ticket/48850 https://fedorahosted.org/389/attachment/ticket/48850/0001-Ticket-48850-Corr… -- Sincerely, William Brown Software Engineer Red Hat, Brisbane

1 0

Please review: 48849, systemd breaks build in rawhide
by William Brown 22 May '16

22 May '16

https://fedorahosted.org/389/ticket/48849 https://fedorahosted.org/389/attachment/ticket/48849/0001-Ticket-48849-Syst… -- Sincerely, William Brown Software Engineer Red Hat, Brisbane

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

389-devel May 2016