I have a RHEL 5.10 system running sssd version sssd-1.5.1-70.el5, SSSD is configured as a client of two IPA servers, config will be at the bottom.
This thing has been chugging along for years at this point without issue, but performance hit the wall this morning and SSSD appears to be the issue.
logins using kerberos are taking an extremely long time, I expect because of user data enumeration, sudo passwords are taking an extremely long time, su passwords etc.
Cranking up debugging is not revealing anything glaring to my untrained eye.
Running an anonymous ldapsearch against the IPA servers (suggested in debug documentation) is not possible as anonymous binds are disabled on the server end. Though the connection does go through enough to indicate that such is the case.
As well restarting SSSD takes a very long time, ~1-2 minutes when compared with other systems on the same platform/hardware (seconds).
Any folks have ideas?
Please CC me as I am subscribed to the digest.
Thanks, -Erinn
Config:
[sssd] config_file_version = 2
# Number of times services should attempt to reconnect in the # event of a crash or restart before they give up reconnection_retries = 3 debug_level = 10
# If a back end is particularly slow you can raise this timeout here sbus_timeout = 30 services = nss, pam
domains = example.com [nss] # The following prevents SSSD from searching for the root user/group in # all domains (you can add here a comma-separated list of system accounts that # are always going to be /etc/passwd users, or that you want to filter out). filter_groups = root,apache filter_users = root,apache reconnection_retries = 3
[pam] reconnection_retries = 3
[domain/example.com] cache_credentials = True krb5_store_password_if_offline = True ipa_domain = example.com id_provider = ipa auth_provider = ipa access_provider = ipa chpass_provider = ipa ipa_server = _srv_, ipa.example.com ldap_tls_cacert = /etc/ipa/ca.crt enumeration = true
On 02/03/2014 02:17 PM, Erinn Looney-Triggs wrote:
I have a RHEL 5.10 system running sssd version sssd-1.5.1-70.el5, SSSD is configured as a client of two IPA servers, config will be at the bottom.
This thing has been chugging along for years at this point without issue, but performance hit the wall this morning and SSSD appears to be the issue.
logins using kerberos are taking an extremely long time, I expect because of user data enumeration, sudo passwords are taking an extremely long time, su passwords etc.
Cranking up debugging is not revealing anything glaring to my untrained eye.
Running an anonymous ldapsearch against the IPA servers (suggested in debug documentation) is not possible as anonymous binds are disabled on the server end. Though the connection does go through enough to indicate that such is the case.
As well restarting SSSD takes a very long time, ~1-2 minutes when compared with other systems on the same platform/hardware (seconds).
Any folks have ideas?
Please CC me as I am subscribed to the digest.
Thanks, -Erinn
Config:
[sssd] config_file_version = 2
# Number of times services should attempt to reconnect in the # event of a crash or restart before they give up reconnection_retries = 3 debug_level = 10
# If a back end is particularly slow you can raise this timeout here sbus_timeout = 30 services = nss, pam
domains = example.com [nss] # The following prevents SSSD from searching for the root user/group in # all domains (you can add here a comma-separated list of system accounts that # are always going to be /etc/passwd users, or that you want to filter out). filter_groups = root,apache filter_users = root,apache reconnection_retries = 3
[pam] reconnection_retries = 3
[domain/example.com] cache_credentials = True krb5_store_password_if_offline = True ipa_domain = example.com id_provider = ipa auth_provider = ipa access_provider = ipa chpass_provider = ipa ipa_server = _srv_, ipa.example.com ldap_tls_cacert = /etc/ipa/ca.crt enumeration = true _______________________________________________ sssd-users mailing list sssd-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users
What is SSSD doing at the start? Enumerating? Is it taking most of the time at start? If so I would think that enumeration is really the issue here.
Do you really need enumeration? If you do may be it makes sense to increase idle timeout and enumeration timeout so that SSSD do not break the connection if idle and do not do enumeration that frequently.
On 02/03/2014 02:17 PM, Erinn Looney-Triggs wrote:
I have a RHEL 5.10 system running sssd version sssd-1.5.1-70.el5, SSSD is configured as a client of two IPA servers, config will be at the bottom.
This thing has been chugging along for years at this point without issue, but performance hit the wall this morning and SSSD appears to be the issue.
logins using kerberos are taking an extremely long time, I expect because of user data enumeration, sudo passwords are taking an extremely long time, su passwords etc.
Cranking up debugging is not revealing anything glaring to my untrained eye.
Running an anonymous ldapsearch against the IPA servers (suggested in debug documentation) is not possible as anonymous binds are disabled on the server end. Though the connection does go through enough to indicate that such is the case.
As well restarting SSSD takes a very long time, ~1-2 minutes when compared with other systems on the same platform/hardware (seconds).
Any folks have ideas?
Please CC me as I am subscribed to the digest.
Thanks, -Erinn
Config:
[sssd] config_file_version = 2
# Number of times services should attempt to reconnect in the # event of a crash or restart before they give up reconnection_retries = 3 debug_level = 10
# If a back end is particularly slow you can raise this timeout here sbus_timeout = 30 services = nss, pam
domains = example.com [nss] # The following prevents SSSD from searching for the root user/group in # all domains (you can add here a comma-separated list of system accounts that # are always going to be /etc/passwd users, or that you want to filter out). filter_groups = root,apache filter_users = root,apache reconnection_retries = 3
[pam] reconnection_retries = 3
[domain/example.com] cache_credentials = True krb5_store_password_if_offline = True ipa_domain = example.com id_provider = ipa auth_provider = ipa access_provider = ipa chpass_provider = ipa ipa_server = _srv_, ipa.example.com ldap_tls_cacert = /etc/ipa/ca.crt enumeration = true _______________________________________________ sssd-users mailing list sssd-users@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/sssd-users
What is SSSD doing at the start? Enumerating? Is it taking most of the time at start? If so I would think that enumeration is really the issue here.
Do you really need enumeration? If you do may be it makes sense to increase idle timeout and enumeration timeout so that SSSD do not break the connection if idle and do not do enumeration that frequently.
On Mon, Feb 03, 2014 at 12:17:10PM -0700, Erinn Looney-Triggs wrote:
I have a RHEL 5.10 system running sssd version sssd-1.5.1-70.el5, SSSD is configured as a client of two IPA servers, config will be at the bottom.
This thing has been chugging along for years at this point without issue, but performance hit the wall this morning and SSSD appears to be the issue.
logins using kerberos are taking an extremely long time, I expect because of user data enumeration, sudo passwords are taking an extremely long time, su passwords etc.
Cranking up debugging is not revealing anything glaring to my untrained eye.
Running an anonymous ldapsearch against the IPA servers (suggested in debug documentation) is not possible as anonymous binds are disabled on the server end. Though the connection does go through enough to indicate that such is the case.
As well restarting SSSD takes a very long time, ~1-2 minutes when compared with other systems on the same platform/hardware (seconds).
Any folks have ideas?
Please CC me as I am subscribed to the digest.
Thanks, -Erinn
What Dmitri said would by my suggestion as well. Try to disable enumeration for a test and see if the startup and general performance improves. Or check the sssd.log for any signs of sssd restarting the backends.
How many users and groups are there in the directory?
On 2/4/2014 1:40 AM, Jakub Hrozek wrote:
On Mon, Feb 03, 2014 at 12:17:10PM -0700, Erinn Looney-Triggs wrote:
I have a RHEL 5.10 system running sssd version sssd-1.5.1-70.el5, SSSD is configured as a client of two IPA servers, config will be at the bottom.
This thing has been chugging along for years at this point without issue, but performance hit the wall this morning and SSSD appears to be the issue.
logins using kerberos are taking an extremely long time, I expect because of user data enumeration, sudo passwords are taking an extremely long time, su passwords etc.
Cranking up debugging is not revealing anything glaring to my untrained eye.
Running an anonymous ldapsearch against the IPA servers (suggested in debug documentation) is not possible as anonymous binds are disabled on the server end. Though the connection does go through enough to indicate that such is the case.
As well restarting SSSD takes a very long time, ~1-2 minutes when compared with other systems on the same platform/hardware (seconds).
Any folks have ideas?
Please CC me as I am subscribed to the digest.
Thanks, -Erinn
What Dmitri said would by my suggestion as well. Try to disable enumeration for a test and see if the startup and general performance improves. Or check the sssd.log for any signs of sssd restarting the backends.
How many users and groups are there in the directory?
Apologies on the delay. It is actually a very small setup < 50 users and groups total.
However, the problem turned out not to be with SSSD but with rsyslog on the system. The product that is deployed on there switched from logging locally to sending logs via syslog. It was sending a large number of logs and things began to stack up pretty badly pretty quickly. For reasons that aren't exactly clear to me this ended up effecting SSSD's performance as well. Nevertheless, switching back to logging locally in our product, SSSD happy, sending via syslog SSSD very unhappy.
Thanks for the suggestions though, I tried both with enumeration and without.
-Erinn
On Fri, Feb 07, 2014 at 09:46:04AM -0700, Erinn Looney-Triggs wrote:
Apologies on the delay. It is actually a very small setup < 50 users and groups total.
However, the problem turned out not to be with SSSD but with rsyslog on the system. The product that is deployed on there switched from logging locally to sending logs via syslog. It was sending a large number of logs and things began to stack up pretty badly pretty quickly. For reasons that aren't exactly clear to me this ended up effecting SSSD's performance as well. Nevertheless, switching back to logging locally in our product, SSSD happy, sending via syslog SSSD very unhappy.
Thanks for the suggestions though, I tried both with enumeration and without.
Interesting, I still suspect disk IO, but I'm glad you got the problem sorted out!
sssd-users@lists.fedorahosted.org