Hi Ludwig,
It's been a while that I've been working on this trying to work through various
details and get more information. Thanks very much for responding, and especially for
such detail and insight into the process. It has helped me dig in more thoroughly and
understand the process better.
So far there doesn't seem to be a good way to get around this without severely
changing our business practices or taking vast rearchitecture leaps. We want to continue
to add ACIs rather than move towards eliminating them. Probably we are going to have to
create subsystems with limited sets of ACIs in each. Would be good to do that with
replication, but I don't think there is a way to keep the root ACI attribute from
replicating.
So, my latest question is:
Can we replicate our two main subtrees, instead of replicating the root directly, and thus
prevent the root ACI attribute from replicating, and then set an individualized ACI
attribute at the root of each replicant to serve different subsets of services?
Thanks very much,
Russ.
On Jul 9, 2013, at 3:08 AM, Ludwig Krispenz <lkrispen(a)redhat.com> wrote:
On 07/09/2013 12:01 AM, Russell Beall wrote:
> Hi Ludwig,
>
> Thanks for responding on this.
>
> After further experimentation and re-applying ACI files from earlier times, I find
that the condition probably has been present the whole time and I just didn't notice
because I was focusing on performance related to our Directory Manager-based scripts.
>
> We have specific ACI rules for each service account we issue. I found that by
commenting out 83 of these rules, I was able to cut down the response time from ~2.5
seconds to ~0.3 seconds. Even further if I limited the returned attributes to a small
set.
yes, since access control is evaluated for each attribute returned reducing the requested
attributes will help (and reducing the number of acis of course helped as well)
>
> I think a key problem with this is that our rules apply to most entries, but then it
is only a very specific userdn which has the "allow" permission at the end of
the rule. Is there any way to turn that around so that the userdn might be looked at
first rather than processing the whole rule only to then find out that it applies to an
irrelevant account?
>
> Is there any documentation on how to best optimize a complex set of ACIs?
no, not that I am aware of, but maybe an understanding of how aci processing works can
help you
- first for an entry all potential acis are collected and since the target of an aci can
extend to teh subtree where it is placed all acis from the entry in question up to the
root suffix are copllected
- while collecting the acis, they are placed in two buckets, one for acis with deny
rules, one for acis with allow rules.
- first the deny acis are processed, as soon as a matching aci is found aci processing
stops
- next the allow list is processed until a matching aci is found
when an aci is evaluated, first check if the target rule covers then entry, then if the
targetattrs include the attr evaluated, then if it contains the correct permission (read,
search,...)
and if the target matches then the bind rule part is evaluated, if it matches the bound
dn (and other rules like dns or ip or ....)
So to optimize aci evaluation, you need to try to minimize the above steps
- if you have a directory tree with many nested subtrees and acis affecting only
subtrees, place them as close as possible to the subtree, so no unrelated acis will be
collected
- always all deny acis will be processed, maybe minimizing denies can help
-if you have service accounts which will have access to only parts of the directory tree,
maybe using macro acis can help to reduce the number of acis
>
> You suggested the logging would show the processing order. I tried placing the most
important rule at the top of the list, but it didn't actually change the processing
(and I didn't really expect that anyway). What is it that you are referring to that
might help me reorder/redesign the processing of the rules?
if you enable aci logging you will get at startup a list of acis and their aci index,
later when evaluating access, there is always something like evaluation xxx deny handles,
xxx allow handles and referencing the index of the aci evaluated, you could then probably
get some insight how many and which acis are processed for each request and in which order
and what finally is the decidng aci.
It will also log if a cached evaluation could be used. Evaluation tries to remember if
the set of acis is the same for the attributes handled and then use the previous
evaluation. Caching cannot be used eg if the acis contain rules depending on the entry, eg
self rules or entry#attr rules.
>
> Appreciate any help you can offer.
>
> Regards,
> Russ.
>
> On Jul 4, 2013, at 6:11 AM, Ludwig Krispenz <lkrispen(a)redhat.com>
> wrote:
>
>> Hi,
>>
>> yes, aci processing can become very expensive if you have a lot of acis are used,
the code tries to do soem optimizations to cache evaluation, but a small change in acis
could prevent the use of cached results. So if you do not have a full set of acis from the
"better" times it will be difficult to tell what led to the changed
performance.
>>
>> The aci code in RHDS and SunDS is very similar, but both have done over the time
bug fixes and attempts to optimize, so for a given set of acis performance could be
different.
>>
>> ACI logging slows down a lot, but it could help you to investigate the usage of
your acis and which acis lead to the decision and which otehr acis are processed before,
so it could help in redesigning your acis, what you probably need to do.
>>
>> Regards,
>> Ludwig
>>
>> On 07/04/2013 12:06 AM, Russell Beall wrote:
>>> I did a lot of work experimenting with 389 for use as a replacement to Sun
SJES. Worked really well when I focused my efforts on the backend processing we do with
Directory Manager, except for a few performance issues which are being addressed in bug
reports.
>>>
>>> I thought sure I had done at least some load testing with service accounts.
The service accounts must go through ACL processing, and we have a lot of ACLs. I'm
not sure if I changed something, or if I just didn't quite test this feature enough,
but now that I am doing more development work with service accounts, I am showing a huge
processing hit taken if a service account is used as opposed to Directory Manager. This
is on the order of a second and a half to respond to a simple base query, versus
instantaneous. Our old SJES servers respond very snappily in comparison for this type of
query.
>>>
>>> CPU usage for a single thread maxes out during the time spent waiting and I/O
wait is zero, so I know that probably the bulk of time is being spent processing the ACLs.
This is especially true if I turn on logging for ACL processing, then it takes a very
long time, with one example taking about 9 minutes.
>>>
>>> It seems to be processing and reprocessing the ACLs many many times over.
>>>
>>> I think I must have changed something or done something wrong because I'm
pretty sure I remember much quicker response times when using a service account in earlier
testing.
>>>
>>> This is with 389-ds-base 1.2.10.14 on RedHat 6.2.
>>>
>>> This was an experimental version downloaded to check out a memory
fragmentation option that was coded in, so maybe I just have a version that was mid ACL
processing changes?
>>>
>>> Thanks for any help,
>>> Russ.
>>>
>>>
>>> --
>>> 389 users mailing list
>>> 389-users(a)lists.fedoraproject.org
>>>
https://admin.fedoraproject.org/mailman/listinfo/389-users
>> --
>> 389 users mailing list
>> 389-users(a)lists.fedoraproject.org
>>
https://admin.fedoraproject.org/mailman/listinfo/389-users
>>
>
> --
> 389 users mailing list
> 389-users(a)lists.fedoraproject.org
>
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users(a)lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/389-users