On 02/05/2018 03:38 PM, Jakub Hrozek wrote:
Hi,
I was helping analyze poor performance and server-side load spikes in an
environment where cluster nodes running sssd were all booted up at the
same time.
It turned out that this meant cache entries were expiring at the same
time and also the LDAP connection was expiring and reconnecting at the
same time. There are some tickets we filed (the ideas were mostly
William's) and I wanted to discuss them here.
https://pagure.io/SSSD/sssd/issue/3623 - Extend object lifetime if the
object hadn't changed in a long time
- I think this the most controversial and as we discussed a bit on
our phone call this is probably too dangerous to do by default.
Nonetheless, for resolving identity requests, it might be a
tunable that might provide a nice performance benefit.
If configurable, it can be done. We may try to be clever here, e.g.
refresh user if group changed.
https://pagure.io/SSSD/sssd/issue/3624 - Randomize cache lifetime by
a
couple of percent
- What the title says. This might prevent hammering the servers in
case the cluster nodes went up at the same time and had the same
expire timestamps for all objects. Again, I'm not sure if this
makes sense by default, because it adds a bit of a fuzzy
behaviour, but I think it makes sense as a configurable.
Yes. This might help a lot in this scenario.
https://pagure.io/SSSD/sssd/issue/3625 - Make sure periodical tasks
use
randomization
- The be_ptask API already supports a bit of randomization, but
we're not really using it. I guess the review should be
case-by-case, but at least for ptasks that fetch any data from the
back end, I would even just randomize a bit by default.
It can be done directly in ptask, enabled by configuration option.
However we must make sure that randomized schedules still make sense
(e.g. sudo smart refresh inteval << sudo full refresh interval).
https://pagure.io/SSSD/sssd/issue/3630 - Randomize
ldap_connection_expire_timeout either by default or w/ a configure option
- Again, if all connections expire and are reconnected at the same
time, the servers suffer.
Agree.
Does anyone have an opinion on the issues? I think at least the
connection timeout is something we should look at, because IIRC that was
causing the most issues on the IDM servers. The other tickets are IMO
less important and I'm even not sure if we should implement them by
default.
There are also two more things we can done:
a) Implement "throttling" behaviour on periodical refresh of expired
object. We can update the object in several batches scheduled over large
(and random) period of time.
b) I do not remember the exact name but we wanted to implement a 389ds
mechanism that would push changes to clients instead of clients pulling
for them. We may want to revisit.