On Tue, Apr 14, 2015 at 03:15:15PM -0400, tmpchq wrote:
started with level 6 and went up to 10 with pretty much the same logs.
Look at the lines around the gap from 13:05:06 to 13:05:08. That's when the delay happened. Responders other than nss and pam did not log anything within that time span.
It is this line that was waiting for response: (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_dom_forwarder] (0x0100): pam_dp_send_req returned 0
----- sssd_nss.log ----- (Tue Apr 14 13:05:06 2015) [sssd[nss]] [check_cache] (0x0400): Cached entry is valid, returning.. (Tue Apr 14 13:05:06 2015) [sssd[nss]] [nss_cmd_getpwnam_search] (0x0400): Returning info for user [qchang@mr.ric] (Tue Apr 14 13:05:06 2015) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f8a5b56cc60][20] (Tue Apr 14 13:05:06 2015) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f8a5b56cc60][20] (Tue Apr 14 13:05:06 2015) [sssd[nss]] [nss_cmd_getbynam] (0x0400): Running command [17] with input [qchang]. (Tue Apr 14 13:05:06 2015) [sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'qchang' matched without domain, user is qchang (Tue Apr 14 13:05:06 2015) [sssd[nss]] [nss_cmd_getbynam] (0x0100): Requesting info for [qchang] from [<ALL>] (Tue Apr 14 13:05:06 2015) [sssd[nss]] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/mr.ric/qchang] (Tue Apr 14 13:05:06 2015) [sssd[nss]] [nss_cmd_getpwnam_search] (0x0100): Requesting info for [qchang@mr.ric] (Tue Apr 14 13:05:06 2015) [sssd[nss]] [ldb] (0x4000): Added timed event "ltdb_callback": 0x7f8a5b56c920
(Tue Apr 14 13:05:06 2015) [sssd[nss]] [ldb] (0x4000): Added timed event "ltdb_timeout": 0x7f8a5b574840
(Tue Apr 14 13:05:06 2015) [sssd[nss]] [ldb] (0x4000): Running timer event 0x7f8a5b56c920 "ltdb_callback"
(Tue Apr 14 13:05:06 2015) [sssd[nss]] [ldb] (0x4000): Destroying timer event 0x7f8a5b574840 "ltdb_timeout"
(Tue Apr 14 13:05:06 2015) [sssd[nss]] [ldb] (0x4000): Ending timer event 0x7f8a5b56c920 "ltdb_callback"
(Tue Apr 14 13:05:06 2015) [sssd[nss]] [check_cache] (0x0400): Cached entry is valid, returning.. (Tue Apr 14 13:05:06 2015) [sssd[nss]] [nss_cmd_getpwnam_search] (0x0400): Returning info for user [qchang@mr.ric] (Tue Apr 14 13:05:06 2015) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f8a5b56cc60][20] (Tue Apr 14 13:05:08 2015) [sssd[nss]] [get_client_cred] (0x4000): Client creds: euid[81] egid[81] pid[503]. (Tue Apr 14 13:05:08 2015) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f8a5b574840][21] (Tue Apr 14 13:05:08 2015) [sssd[nss]] [accept_fd_handler] (0x0400): Client connected! (Tue Apr 14 13:05:08 2015) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f8a5b574840][21] (Tue Apr 14 13:05:08 2015) [sssd[nss]] [sss_cmd_get_version] (0x0200): Received client version [1]. (Tue Apr 14 13:05:08 2015) [sssd[nss]] [sss_cmd_get_version] (0x0200): Offered version [1]. (Tue Apr 14 13:05:08 2015) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f8a5b574840][21] (Tue Apr 14 13:05:08 2015) [sssd[nss]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f8a5b574840][21] (Tue Apr 14 13:05:08 2015) [sssd[nss]] [nss_cmd_getbynam] (0x0400): Running command [38] with input [root]. (Tue Apr 14 13:05:08 2015) [sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'root' matched without domain, user is root (Tue Apr 14 13:05:08 2015) [sssd[nss]] [nss_cmd_getbynam] (0x0100): Requesting info for [root] from [<ALL>] (Tue Apr 14 13:05:08 2015) [sssd[nss]] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/mr.ric/root] (Tue Apr 14 13:05:08 2015) [sssd[nss]] [nss_cmd_initgroups_search] (0x0400): User [root] does not exist in [mr.ric]! (negative cache) (Tue Apr 14 13:05:08 2015) [sssd[nss]] [nss_cmd_initgroups_search] (0x0080): No matching domain found for [root], fail!
----- sssd_pam.log ----- (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_check_user_search] (0x0400): Returning info for user [qchang@mr.ric] (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_dp_send_req] (0x0100): Sending request with the following data: (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): command: PAM_ACCT_MGMT (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): domain: mr.ric (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): user: qchang (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): service: sshd (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): tty: ssh (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): ruser: not set (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): rhost: 172.25.12.157 (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): authtok type: 0 (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): newauthtok type: 0 (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): priv: 1 (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): cli_pid: 7402 (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_print_data] (0x0100): logon name: qchang (Tue Apr 14 13:05:06 2015) [sssd[pam]] [sbus_add_timeout] (0x2000): 0x7f1aff4200a0 (Tue Apr 14 13:05:06 2015) [sssd[pam]] [pam_dom_forwarder] (0x0100): pam_dp_send_req returned 0 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [sbus_remove_timeout] (0x2000): 0x7f1aff4200a0 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [sbus_dispatch] (0x4000): dbus conn: 0x7f1aff41ddd0 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [sbus_dispatch] (0x4000): Dispatching. (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_dp_process_reply] (0x0100): received: [0][mr.ric] (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_reply] (0x0200): pam_reply called with result [0]. (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_reply] (0x0200): blen: 23 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f1aff421b40][19] (Tue Apr 14 13:05:08 2015) [sssd[pam]] [reset_idle_timer] (0x4000): Idle timer re-set for client [0x7f1aff421b40][19] (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_cmd_setcred] (0x0100): entering pam_cmd_setcred (Tue Apr 14 13:05:08 2015) [sssd[pam]] [sss_parse_name_for_domains] (0x0200): name 'qchang' matched without domain, user is qchang (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): command: PAM_SETCRED (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): domain: not set (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): user: qchang (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): service: sshd (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): tty: ssh (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): ruser: not set (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): rhost: 172.25.12.157 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): authtok type: 0 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): newauthtok type: 0 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): priv: 1 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): cli_pid: 7402 (Tue Apr 14 13:05:08 2015) [sssd[pam]] [pam_print_data] (0x0100): logon name: qchang (Tue Apr 14 13:05:08 2015) [sssd[pam]] [sss_ncache_check_str] (0x2000): Checking negative cache for [NCE/USER/mr.ric/qchang]
The logs are incomplete, moreover we also need the back end logs to be sure.
From the snippet you sent, I only see the account phase is taking 2
seconds. It's impossible what exactly is taking long without the domain logs.