I have gone through the log and got the following sections for clarity.
Red lines are where the pause/delay happened. It looks to me two of them (PAM_AUTHENTICATE and Retrieving SELinux user mapping) are due to SSSD taking a break (All data has been sent!) from talking to the IPA server.
I have done some ping test among the servers, average ping time is 0.5ms.
----- one second for PAM_AUTHENTICATE ----- (Wed Apr 15 14:19:44:122152 2015) [sssd[be[mr.ric]]] [be_pam_handler] (0x0100): Got request with the following data (Wed Apr 15 14:19:44:122163 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): command: PAM_AUTHENTICATE (Wed Apr 15 14:19:44:122173 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): domain: mr.ric (Wed Apr 15 14:19:44:122183 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): user: qchang (Wed Apr 15 14:19:44:122193 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): service: sshd (Wed Apr 15 14:19:44:122202 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): tty: ssh (Wed Apr 15 14:19:44:122212 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): ruser: (Wed Apr 15 14:19:44:122221 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): rhost: 172.25.12.157 (Wed Apr 15 14:19:44:122232 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): authtok type: 1 (Wed Apr 15 14:19:44:122242 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): newauthtok type: 0 (Wed Apr 15 14:19:44:122251 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): priv: 1 (Wed Apr 15 14:19:44:122261 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): cli_pid: 28330 (Wed Apr 15 14:19:44:122271 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): logon name: not set (Wed Apr 15 14:19:44:122399 2015) [sssd[be[mr.ric]]] [fo_resolve_service_send] (0x0100): Trying to resolve service 'IPA' (Wed Apr 15 14:19:44:122422 2015) [sssd[be[mr.ric]]] [resolve_srv_send] (0x0200): The status of SRV lookup is resolved (Wed Apr 15 14:19:44:122443 2015) [sssd[be[mr.ric]]] [be_resolve_server_process] (0x0200): Found address for server mripa1.mr.ric: [172.25.12.153] TTL 1200 (Wed Apr 15 14:19:44:122460 2015) [sssd[be[mr.ric]]] [ipa_resolve_callback] (0x0400): Constructed uri 'ldap://mripa1.mr.ric' (Wed Apr 15 14:19:44:123509 2015) [sssd[be[mr.ric]]] [write_pipe_handler] (0x0400): All data has been sent! (Wed Apr 15 14:19:45:169789 2015) [sssd[be[mr.ric]]] [child_sig_handler] (0x0100): child [28332] finished successfully. (Wed Apr 15 14:19:45:169853 2015) [sssd[be[mr.ric]]] [read_pipe_handler] (0x0400): EOF received, client finished
----- one second for PAM_ACCT_MGMT ----- (Wed Apr 15 14:19:45:225894 2015) [sssd[be[mr.ric]]] [be_pam_handler] (0x0100): Got request with the following data (Wed Apr 15 14:19:45:225905 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): command: PAM_ACCT_MGMT (Wed Apr 15 14:19:45:225915 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): domain: mr.ric (Wed Apr 15 14:19:45:225925 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): user: qchang (Wed Apr 15 14:19:45:225934 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): service: sshd (Wed Apr 15 14:19:45:225944 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): tty: ssh (Wed Apr 15 14:19:45:225954 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): ruser: (Wed Apr 15 14:19:45:225963 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): rhost: 172.25.12.157 (Wed Apr 15 14:19:45:225973 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): authtok type: 0 (Wed Apr 15 14:19:45:225983 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): newauthtok type: 0 (Wed Apr 15 14:19:45:225992 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): priv: 1 (Wed Apr 15 14:19:45:226002 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): cli_pid: 28330 (Wed Apr 15 14:19:45:226012 2015) [sssd[be[mr.ric]]] [pam_print_data] (0x0100): logon name: not set (Wed Apr 15 14:19:45:226030 2015) [sssd[be[mr.ric]]] [sdap_access_send] (0x0400): Performing access check for user [qchang] (Wed Apr 15 14:19:45:226168 2015) [sssd[be[mr.ric]]] [sdap_account_expired_rhds] (0x0400): Performing RHDS access check for user [qchang] (Wed Apr 15 14:19:45:226213 2015) [sssd[be[mr.ric]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(objectClass=ipaHost)(fqdn= mripadm.mr.ric))][cn=accounts,dc=mr,dc=ric]. (Wed Apr 15 14:19:45:228995 2015) [sssd[be[mr.ric]]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set (Wed Apr 15 14:19:45:229039 2015) [sssd[be[mr.ric]]] [sdap_has_deref_support] (0x0400): The server supports deref method OpenLDAP (Wed Apr 15 14:19:45:229063 2015) [sssd[be[mr.ric]]] [sdap_x_deref_search_send] (0x0400): Dereferencing entry [fqdn=mripadm.mr.ric,cn=computers,cn=acc ounts,dc=mr,dc=ric] using OpenLDAP deref (Wed Apr 15 14:19:45:229077 2015) [sssd[be[mr.ric]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [no filter][fqdn=mripadm.mr.ri c,cn=computers,cn=accounts,dc=mr,dc=ric]. (Wed Apr 15 14:19:46:175840 2015) [sssd[be[mr.ric]]] [sdap_x_deref_parse_entry] (0x0400): Got deref control (Wed Apr 15 14:19:46:175969 2015) [sssd[be[mr.ric]]] [sdap_x_deref_parse_entry] (0x0400): All deref results from a single control parsed (Wed Apr 15 14:19:46:176072 2015) [sssd[be[mr.ric]]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set
----- one second for Retrieving SELinux user mapping ----- (Wed Apr 15 14:19:46:204860 2015) [sssd[be[mr.ric]]] [ipa_get_selinux_send] (0x0400): Retrieving SELinux user mapping (Wed Apr 15 14:19:46:204893 2015) [sssd[be[mr.ric]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(cn=ipaConfig)(objectClass= ipaGuiConfig))][cn=etc,dc=mr,dc=ric]. (Wed Apr 15 14:19:46:206852 2015) [sssd[be[mr.ric]]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set (Wed Apr 15 14:19:46:206899 2015) [sssd[be[mr.ric]]] [ipa_selinux_get_maps_next] (0x0400): Trying to fetch SELinux maps with following parameters: [2] [(&(objectclass=ipaselinuxusermap)(ipaEnabledFlag=TRUE))][cn=selinux,dc=mr,dc=ric] (Wed Apr 15 14:19:46:206917 2015) [sssd[be[mr.ric]]] [sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with [(&(objectclass=ipaselinuxuser map)(ipaEnabledFlag=TRUE))][cn=selinux,dc=mr,dc=ric]. (Wed Apr 15 14:19:46:207979 2015) [sssd[be[mr.ric]]] [sdap_get_generic_op_finished] (0x0400): Search result: Success(0), no errmsg set (Wed Apr 15 14:19:46:208011 2015) [sssd[be[mr.ric]]] [ipa_selinux_get_maps_done] (0x0400): No SELinux user maps found! (Wed Apr 15 14:19:46:210900 2015) [sssd[be[mr.ric]]] [write_pipe_handler] (0x0400): All data has been sent! (Wed Apr 15 14:19:47:557404 2015) [sssd[be[mr.ric]]] [be_get_account_info] (0x0200): Got request for [0x1003][1][name=avahi] (Wed Apr 15 14:19:47:557429 2015) [sssd[be[mr.ric]]] [be_req_set_domain] (0x0400): Changing request domain from [mr.ric] to [mr.ric] -----
----- 0.01 second for lookup for avahi and poltitd combined ----- (Wed Apr 15 14:19:47:557404 2015) [sssd[be[mr.ric]]] [be_get_account_info] (0x0200): Got request for [0x1003][1][name=avahi] ... (Wed Apr 15 14:19:47:569055 2015) [sssd[be[mr.ric]]] [ipa_id_get_account_info_orig_done] (0x0080): Object not found, ending request (Wed Apr 15 14:19:47:569093 2015) [sssd[be[mr.ric]]] [acctinfo_callback] (0x0100): Request processed. Returned 3,0,Account info lookup failed (Wed Apr 15 14:19:47:570494 2015) [sssd[be[mr.ric]]] [be_get_account_info] (0x0200): Got request for [0x1003][1][name=polkitd] ... (Wed Apr 15 14:19:47:581941 2015) [sssd[be[mr.ric]]] [ipa_id_get_account_info_orig_done] (0x0080): Object not found, ending request (Wed Apr 15 14:19:47:581975 2015) [sssd[be[mr.ric]]] [acctinfo_callback] (0x0100): Request processed. Returned 3,0,Account info lookup failed -----
On Wed, Apr 15, 2015 at 2:36 PM, tmpchq tmpchq@gmail.com wrote:
I noticed that when login to IPA server (mripa1) itself, the only PAM process is PAM_OPEN_SESSION, whereas for mripadm it goes through 5: PAM_AUTHENTICATE, PAM_ACCT_MGMT, PAM_SETCRED, PAM_OPEN_SESSION, PAM_SETCRED in that order. I have virtually identical SSSD and PAM config for both servers...