Over time, I’ve been having seemingly random sssd quits that I’ve not been able to figure out. Today, I finally traced it to fluctuations on my Infiniband fabric:
sssd.log
(Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16 (Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface idx 4 (ib0) flags 0x1003 (broadcast,multicast,up) (Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16 (Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface idx 4 (ib0) flags 0x11043 (broadcast,multicast,up,running,lower)
This exactly corresponds to the time in /var/log/messages for the unexplained shutdown:
2015-11-03T13:17:59-05:00 node75 sssd[pam]: Shutting down 2015-11-03T13:17:59-05:00 node75 sssd[be[default]]: Shutting down 2015-11-03T13:17:59-05:00 node75 sssd[nss]: Shutting down
Here is sssd_default.log for good measure:
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1414770/0x14133d0 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1414770/0x13fef90 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_ptask_destructor] (0x0400): Terminating periodic task [Cleanup of default] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sdap_handle_release] (0x2000): Trace: sh[0x14bd850], connected[1], ops[(nil)], ldap[0x1424260], destructor_lock[0], release_memory[0] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_connection_callback] (0x4000): Successfully removed connection callback. (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1415970/0x1416430 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): The symlink points to [/var/lib/sss/pipes/private/sbus-dp_default.18702] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): The path including our pid is [/var/lib/sss/pipes/private/sbus-dp_default.18702] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): Removed the symlink (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_client_destructor] (0x0400): Removed PAM client (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_client_destructor] (0x0400): Removed NSS client
I can duplicate this by manually taking down the Infiniband link:
[root@node24 ~]# service sssd status sssd (pid 9132) is running... [root@node24 ~]# ifdown ib0 [root@node24 ~]# service sssd status sssd dead but pid file exists
I have also noticed that sssd will not start on boot. As I know that Infiniband tends to flutter a little bit before the link comes up, I’m thinking this is probably the same cause.
Can anyone explain this behavior and tell me what I might do to prevent it?
-- ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* || \UTGERS |---------------------*O*--------------------- ||_// Biomedical | Ryan Novosielski - Senior Technologist || \ and Health | novosirj@rutgers.edu - 973/972.0922 (2x0922) || \ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark `'