On 07/28/2016 11:17 AM, Lukas Slebodnik wrote:
On (28/07/16 10:24), thierry bordaz wrote:
>
> On 07/28/2016 09:39 AM, Jakub Hrozek wrote:
>> On Wed, Jul 27, 2016 at 04:09:07PM +0200, thierry bordaz wrote:
>>> On 07/27/2016 03:36 PM, Jakub Hrozek wrote:
>>>> On Wed, Jul 27, 2016 at 02:55:37PM +0200, thierry bordaz wrote:
>>>>> On 07/27/2016 01:56 PM, Jakub Hrozek wrote:
>>>>>> On Wed, Jul 27, 2016 at 01:03:59PM +0200, Jakub Hrozek wrote:
>>>>>>> On Wed, Jul 27, 2016 at 12:22:46PM +0200, Lukas Slebodnik
wrote:
>>>>>>>> On (27/07/16 12:08), Jakub Hrozek wrote:
>>>>>>>>> On Wed, Jul 27, 2016 at 12:02:24PM +0200, Jakub
Hrozek wrote:
>>>>>>>>>> On Wed, Jul 27, 2016 at 11:54:16AM +0200, Lukas
Slebodnik wrote:
>>>>>>>>>>> ehlo,
>>>>>>>>>>>
>>>>>>>>>>> attached patch fixes acces denied after
activating user in 389ds.
>>>>>>>>>>> Jakub had some comments/ideas in ticket but I
think it's better to discuss
>>>>>>>>>>> about virtual attributes and timestamp cache
on mailing list.
>>>>>>>>>> Yes, so the comment I have is that while this
works, it might break some
>>>>>>>>>> strange LDAP servers.
>>>>>>>>>>
>>>>>>>>>> We use modifyTimestamp as a 'positive'
indicator that the entry has not
>>>>>>>>>> changed -- if the modifyTimestamp didn't
change, we consider the cached
>>>>>>>>>> entry the same as what is on the server and only
bump the timestamp
>>>>>>>>>> cache. If the timestamp is different, we do a
deep-comparison of cached
>>>>>>>>>> attribute values with what is on the LDAP server
and write the sysdb
>>>>>>>>>> cache entry only if the attributes differ.
>>>>>>>>>>
>>>>>>>>>> I was wondering if we can use the modifyTimestamp
at all, then, because
>>>>>>>>>> even if it's the same, we might want to check
the attributes to see if
>>>>>>>>>> some of the values are different because some of
the attributes might be
>>>>>>>>>> this operational/virtual attribute..
>>>>>>>>> Sorry, sent too soon.
>>>>>>>>>
>>>>>>>>> I think the questions are -- 1) can we enumerate the
virtual attributes?
>>>>>>>> That might be a question for 389-ds developers.
>>>>>>>> But it's very likely it will be different on other
LDAP servers.
>>>>>>>>
>>>>>>>>> 2) Would different LDAP servers have different
virtual attributes.
>>>>>>>>>
>>>>>>>>> For 2) maybe a possible solution might be to set a
non-existing
>>>>>>>>> modifyTimestamp attribute value, but I would consider
that only a
>>>>>>>>> kludge, we shouldn't break existing setups..
>>>>>>>> I am not satisfied with this POC solution either.
>>>>>>>>
>>>>>>>> So should we remove usage of modifyTimestamp for
detecting changes?
>>>>>>> I would prefer to ask the DS developers before removing it
completely.
>>>>>>>
>>>>>>> At least for large groups it might take a long time to
compare all attribute
>>>>>>> values and IIRC we don't depend on any virtual attributes
for groups. Maybe
>>>>>>> we could parametrize that part of the code and enable the
fast way with
>>>>>>> modifyTimestamps for 'known' server types, that is
for setups with AD and
>>>>>>> IPA providers.
>>>>>>>
>>>>>>> For users, there is typically not as many attributes so we
should be
>>>>>>> fine deep-comparing all attributes.
>>>>>> I'm adding Thierry (so please reply-to-all to keep him in the
thread).
>>>>>>
>>>>>> Thierry, in the latest sssd version we tried to add a
performance
>>>>>> improvement related to how we store SSSD entries in the cache.
The short
>>>>>> version is that we store the modifyTimestamp attribute in the
cache and
>>>>>> when we fetch an entry, we compare the entry modifyTimestamp with
what
>>>>>> is on the server. When the two are the same, we say that the
entry did
>>>>>> not change and don't update the cache.
>>>>>>
>>>>>> This works fine for most attributes, but not for attributes like
>>>>>> nsAccountLock which do not change modifyTimestamp when they are
>>>>>> modified. So when an entry was already cached but then
nsAccountLock
>>>>>> changed, we treated the entry as the same and never read the new
>>>>>> nsAccountLock value.
>>>>>>
>>>>>> To fix this, I think we have several options:
>>>>>> 1) special-case the nsAccountLock. This seems a bit
dangerous,
>>>>>> because I'm not sure we can say that some other
attribute we are
>>>>>> interested in behaves the same as nsAccountLock.
>>>>>> 2) drop the modifyTimestamp optimization completely. Then
we fall
>>>>>> back to comparing the attribute values, which might work,
but for
>>>>>> huge objects like groups with thousands of members, this
might be
>>>>>> too expensive.
>>>>>> 3) only use the modifyTimestamp optimization for cases
where we know
>>>>>> we don't read any virtual attributes.
>>>>>>
>>>>>> And my question is -- can we, in general, know if the
modifyTimestamp
>>>>>> way of detecting changes is realiable for all LDAP servers? Or do
you
>>>>>> think it should only be used for cases where we know we are not
>>>>>> interested in any virtual attributes (that would mostly be
storing
>>>>>> groups from servers where we know exactly what is on the server
side,
>>>>>> like IPA or AD).
>>>>> Hello,
>>>>>
>>>>> Relying on modifytimestamp looks a good idea. Any MOD/MODRDN will
update it,
>>>>> except I think it is unchanged when updating some password policy
>>>>> attributes.
>>>>>
>>>>> Regarding virtual attribute, the only one I know in IPA is
nsaccountlock.
>>>>> nsaccountlock is an operational attribute (you need to request it to
see it)
>>>>> and is also a virtual attribute BUT only for 'staged' and
'deleted' users.
>>>>> It is a stored attribute for regular users and we should update
>>>>> modifytimestamp when it is set.
>>>>>
>>>>> thanks
>>>>> thierry
>>>> OK, in that case it seems like we can special-case it. But do you know
>>>> about any other attributes in any other LDAP servers?
>>> Any LDAP server following standard should provide modifytimestamp that
>>> reflect the last update of the entry. Now virtual attribute values may be
>>> "attached" to the entry and its value change without modification
of
>>> modifytimestamp.
>>> For 389-ds and IPA it is fine as virtual value of nsaccountlock is changed
>>> only when the DN change.
>>> For others LDAP servers I suppose it exists the same ability to define
>>> service providers that return virtual attribute values. The difficulty is
>>> that the schema may not give any hint if the retrieved attributes values
>>> were stored or computed and consequently trust modifytimestamp to know if
>>> the values changed or not.
>>> For example in ODSEE, memberof is a virtual attribute.
>> Thank you, for the explanation Thierry.
>>
>> Then to be on the safe side I propose:
>> 1) We add an (probably undocumented?) flag that says whether to use
>> modifyTimestamp to detect entry changes or not
>> 2) for the generic LDAP provider we always really compare the
>> attribute values, in other words the option would be set to
>> false. If there is anyone with performance issues with a generic
>> setup, we tell them to flip the option.
>> 3) For the IPA and AD providers, we set this option to true and use
>> the modifyTimestamp value to detect changes
>> 4) We special case nsAccountLock
> I am not sure I understand why nsAccountLock is a special case ?
> In IPA/389:
>
> * Stage user, it is always 'nsaccountlock: True'
> When stage entry is created or updated, 'modifytimestamp' is also
> updated. So you can rely on modifytimestamp to detect a change in a
> stage user.
> There is no way to update nsaccountlock for a stage entry
> * Deleted user. Idem Stage user
We haven't tested stage users yet. I ran just sssd related regression tests.
> * Regular user. 'nsaccountlock' is *not* a virtual attribute, so if it
> is enable/disable you can rely on modifytimestamp to detect a change
> of 'nsaccountlock' for a regular user.
> Also any change on regular user will update 'modifytimestamp' so you
> can rely on it to detect a change.
>
My experience with ns-inactivate.pl and ns-activate.pl is different.
modifytimestamp is not changed even though nsaccountlock was changed.
It was a plain 389-ds with id_provider ldap in sssd
LS
Hi Lukas,
It may have change recently because I can not reproduce. What versions
are you running ?
#
#initial entry
#
ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w xxx -b
"cn=accounts, <suffix>" 'uid=user0' nsaccountlock
modifytimestamp
createtimestamp
dn: uid=user0,cn=users,cn=accounts,<suffix>
modifytimestamp: *20160728103441Z*
createtimestamp: 20160726160436Z
#
# Inactivate: modifytimestamp changed
#
/usr/sbin/ns-inactivate.pl -Z <instance> -D "cn=directory manager"
-w xxx -I uid=user0,cn=users,cn=accounts,<suffix>
uid=user0,cn=users,cn=accounts,<suffix> inactivated.
ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w xxx -b
"cn=accounts, <suffix>" 'uid=user0' nsaccountlock
modifytimestamp
createtimestamp
dn: uid=user0,cn=users,cn=accounts,<suffix>
nsaccountlock: true
modifytimestamp: *20160728103642Z*
createtimestamp: 20160726160436Z
#
# activate: modifytimestamp changed
#
/usr/sbin/ns-activate.pl -Z <instance> -D "cn=directory manager" -w
xxx -I uid=user0,cn=users,cn=accounts,<suffix>
uid=user0,cn=users,cn=accounts,<suffix> activated.
ldapsearch -LLL -o ldif-wrap=no -D "cn=directory manager" -w xxx -b
"cn=accounts, <suffix>" 'uid=user0' nsaccountlock
modifytimestamp
createtimestamp
dn: uid=user0,cn=users,cn=accounts,<suffix>
modifytimestamp: *20160728103711Z*
createtimestamp: 20160726160436Z