URL:
https://github.com/SSSD/sssd/pull/5552
Title: #5552: files: split update into batches
elkoniu commented:
"""
I modified this PR by adding initial zero step which only starts database transaction and
returns to tevent loop:
https://github.com/elkoniu/sssd/tree/file-batching-zero-step
End results is that DBUS commands can be send before "heavy" calculations
starts.
Please take a look at updated test results (previous test result included):
Estimated time needed to mark domain as inconsistent:
|Version|30 users|30 + 100 users|30 + 250 users|30 + 500 users|
|:---:|:---:|:---:|:---:|:---:|
|Vanilla|91ms|190ms|444ms|891ms|
|Batching|34ms|63ms|105ms|188ms|
|Zero step|1,1ms|1,4ms|0,8ms|1,1ms|
Time consumed by calling operations on `X` users:
```
time for N in `seq X`; do useradd test-user-$N; done
time for N in `seq X`; do userdel -rf test-user-$N; done
```
|Version|100 useradd|100 userdel|250 useradd|250 userdel|500 useradd|500 userdel|
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|Vanilla|0m43.464s|0m41.084s|2m46.462s|2m46.462s|9m55.823s|9m15.531s|
|Batching|0m53.304s|0m48.572s|3m42.061s|4m16.405s|20m15.965s|18m1.768s|
|Zero step|0m56.830s|0m53.676s|3m48.812s|3m46.155s|15m27.024s|14m56.387s|
Conclusion: With dummy zero steep this PR will be able to solve
https://github.com/SSSD/sssd/issues/5091 by marking domain as inconsistent fast enough.
```
==============================================================================================================
[BATCH WITH ZERO STEP]
==============================================================================================================
[30 entries in /etc/passwd]
# Counting users in passwd
[root(a)master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l
30
# Cache cleaning and daemon restart
[root(a)master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf
/var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions
3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation
./sssd_files.log:1743:(2021-04-06 10:27:35:422398): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:156:(2021-04-06 10:27:35:423142): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
423142 - 422398 = 744
./sssd_files.log:2477:(2021-04-06 10:28:02:358816): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:184:(2021-04-06 10:28:02:360463): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
360463 - 358816 = 1647
./sssd_files.log:3211:(2021-04-06 10:28:21:213798): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:212:(2021-04-06 10:28:21:214862): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
214862 - 213798 = 1064
Average = (744 + 1647 + 1064) / 3 = 1152us = 1,152ms
--------------------------------------------------------------------------------------------------------------
[30 + 100 entries in /etc/passwd]
# Adding users
[root(a)master.client.vm /var/log/sssd]# time for N in `seq 100`; do useradd test-user-$N;
done
real 0m56.830s
user 0m6.624s
sys 0m3.577s
# Counting users in passwd
[root(a)master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l
130
# Cache cleaning and daemon restart
[root(a)master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf
/var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions
3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation
./sssd_files.log:5203:(2021-04-06 10:31:42:194803): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:160:(2021-04-06 10:31:42:195815): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
195815 - 194803 = 1012
./sssd_files.log:7667:(2021-04-06 10:32:03:338062): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:188:(2021-04-06 10:32:03:339720): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
339720 - 338062 = 1658
./sssd_files.log:10131:(2021-04-06 10:32:15:219538): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:216:(2021-04-06 10:32:15:221051): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
221051 - 219538 = 1513
Average = (1012 + 1658 + 1513) / 3 = 1394us = 1,394ms
# Deleting users
[root(a)master.client.vm /var/log/sssd]# time for N in `seq 100`; do userdel -rf
test-user-$N; done
real 0m53.676s
user 0m6.299s
sys 0m3.504s
--------------------------------------------------------------------------------------------------------------
[30 + 250 entries in /etc/passwd]
# Adding users
[root(a)master.client.vm /var/log/sssd]# time for N in `seq 250`; do useradd test-user-$N;
done
real 3m48.812s
user 0m28.150s
sys 0m14.616s
# Counting users in passwd
[root(a)master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l
280
# Cache cleaning and daemon restart
[root(a)master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf
/var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions
3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation
./sssd_files.log:10393:(2021-04-06 10:40:16:284891): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:164:(2021-04-06 10:40:16:285288): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
285288 - 284891 = 397
./sssd_files.log:15452:(2021-04-06 10:41:54:614785): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:192:(2021-04-06 10:41:54:616448): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
616448 - 614785 = 1663
./sssd_files.log:20511:(2021-04-06 10:42:14:447177): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:220:(2021-04-06 10:42:14:447551): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
447551 - 447177 = 374
Average = (397 + 1663 + 374) / 3 = 811us = 0,811ms
# Deleting users
[root(a)master.client.vm /var/log/sssd]# time for N in `seq 250`; do userdel -rf
test-user-$N; done
real 3m46.155s
user 0m27.973s
sys 0m14.632s
--------------------------------------------------------------------------------------------------------------
[30 + 500 entries in /etc/passwd]
# Adding users
[root(a)master.client.vm /var/log/sssd]# time for N in `seq 500`; do useradd test-user-$N;
done
real 15m27.024s
user 1m48.542s
sys 0m55.511s
# Counting users in passwd
[root(a)master.client.vm /var/log/sssd]# cat /etc/passwd | wc -l
530
# Cache cleaning and daemon restart
[root(a)master.client.vm /var/log/sssd]# sss_cache -E; systemctl stop sssd; rm -rf
/var/lib/sss/db/*; rm -rf /var/log/sssd/*; systemctl start sssd
# Triggering actions
3 x manual call "touch /etc/passwd" with few seconds in between calls
# Logs timestamp calculation
./sssd_files.log:19043:(2021-04-06 11:07:34:234339): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:156:(2021-04-06 11:07:34:234736): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
234736 - 234339 = 397
./sssd_files.log:28427:(2021-04-06 11:07:49:242399): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:184:(2021-04-06 11:07:49:244265): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
244265 - 242399 = 1866
./sssd_files.log:37811:(2021-04-06 11:08:04:169746): [be[files]] [sf_passwd_cb] (0x0200):
>>>PASSWD_MODIFIED_CB
./sssd_nss.log:212:(2021-04-06 11:08:04:170919): [nss] [sss_resp_domain_inconsistent]
(0x0200): >>>DOMAIN_INCONSISTENT_RECEIVED
170919 - 169746 = 1173
Average = (397 + 1866 + 1173) / 3 = 1145us = 1,145ms
# Deleting users
[root(a)master.client.vm /var/log/sssd]# time for N in `seq 500`; do userdel -rf
test-user-$N; done
real 14m56.387s
user 1m47.032s
sys 0m55.798s
```
"""
See the full comment at
https://github.com/SSSD/sssd/pull/5552#issuecomment-814043592