I have further noticed that the difference is coming from additional calls from sssd to
ldap (from a tcpdump)
It looks like in both cases caching is not working as for every sudo -u user
"cmd" a call is being made to ldap however in the slower case it is making
additional calls (I can't see what they are because they are all tls/ldap)
I am unsure if this is expected behavior ?
I am going to try tweak the sudo rule caching options.