[PATCH] Improve sysdb write performance - sssd-devel - Fedora mailing-lists

3 Jun 2016


      Hi,
the attached patchset improves SSSD by performance by adding a second
ldb cache file that is opened with LDB_FLG_NOSYNC and contains empeheral
data like timestamps. The reason is to avoid updating the sync-cache in
case nothing in fact changed and SSSD is only refreshing the cache to
add new timestamps. Especially for group lookups, this can improve
performance a bit. The first lookup, or any where the actual data
changes is still slow, though.
I don't have a CI link yet, because I'm having issues installing pyldb on
our test machines.
The last patch doesn't touch sysdb, but the LDAP provider. I hope it's
fine to include it in the same set, though.
The full design document is at:
    https://fedorahosted.org/sssd/wiki/DesignDocs/OneFourteenPerformanceImprovem...
I tried to keep the operations contained to the sysdb interface itself,
no provider or responder changes were required.
To test, you can use the systemtap scripts I sent earlier to the list.
For my test user who is a member of about ~200 groups, each containing
several hundred members, running 'id' with an empty cache takes:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Total run time of id was: 8581 ms
Number of zero-level cache transactions: 166
Time spent in level-0 sysdb transactions: 6045 ms
Time spent writing to LDB: 2550 ms
Number of LDAP searches: 326
Time spent waiting for LDAP: 1780 ms
LDAP searches breakdown:
        Number of user requests: 1
        Time spent in user requests: 20
Number of group requests: 162
        Time spent in group requests: 7965
Number of initgroups requests: 1
        Time spent in initgroups requests: 115
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Without the patches, the updates take about the same time as filing the
empty cache. With the patches, when the cache is expired with "sss_cache
-E" and refreshed:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Total run time of id was: 3934 ms
Number of zero-level cache transactions: 165
Time spent in level-0 sysdb transactions: 1404 ms
Time spent writing to LDB: 6 ms
Number of LDAP searches: 326
Time spent waiting for LDAP: 1756 ms
LDAP searches breakdown:
        Number of user requests: 1
        Time spent in user requests: 3
Number of group requests: 162
        Time spent in group requests: 3377
Number of initgroups requests: 1
        Time spent in initgroups requests: 39
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
With the patches, the group lookups performance analysis looks like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Time spent in group sssd_be searches: 3570
Time spent in sdap_nested_group_send/recv: 1609 ms (ratio: 45.07%)
Time spent in zero-level sysdb transactions: 1694 ms (ratio: 47.45%)
Breakdown of sdap_nested_group req (total: 1609 ms)
        sdap_nested_group_process req: 1608
                sdap_nested_group_process_split req: 152
                        sdap_nested_group_check_cache: 147
                                sdap_nested_group_sysdb_search_users: 94
                                sdap_nested_group_sysdb_search_groups: 49
                ldap request breakdown of total 1405
                        sdap_nested_group_deref req: 1455
                                sdap_deref_search_send req 1405
                                processing deref results: 50
                        sdap_nested_group_lookup_user req: 0
                        sdap_nested_group_lookup_group req: 0
                        Time spent refreshing unknown members: 0
Breakdown of results processing (total 1694)
        Time spent populating nested members: 1454
                Time spent searching ldb while populating nested members: 1328
        Time spent saving nested members: 145
        Time spent writing to the ldb: 9 ms
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Apparently, we still can improve the sdap_async.c module to do fewer cache
lookups and optimize the dereference results processing to not parse all
received attributes if the modifyTimestamp is the same as we already have
stored in the cache. But I think the attached patchset already improves
the situation quite a bit and can be reviewed on its own.
So far I tested IPA client and AD client. I still need to test the IPA
server itself and run downstream tests. But I don't think this needs to
block the review.