Over time, I’ve been having seemingly random sssd quits that I’ve not been able to figure out. Today, I finally traced it to fluctuations on my Infiniband fabric:
sssd.log
(Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16 (Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface idx 4 (ib0) flags 0x1003 (broadcast,multicast,up) (Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16 (Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface idx 4 (ib0) flags 0x11043 (broadcast,multicast,up,running,lower)
This exactly corresponds to the time in /var/log/messages for the unexplained shutdown:
2015-11-03T13:17:59-05:00 node75 sssd[pam]: Shutting down 2015-11-03T13:17:59-05:00 node75 sssd[be[default]]: Shutting down 2015-11-03T13:17:59-05:00 node75 sssd[nss]: Shutting down
Here is sssd_default.log for good measure:
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1414770/0x14133d0 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1414770/0x13fef90 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_ptask_destructor] (0x0400): Terminating periodic task [Cleanup of default] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sdap_handle_release] (0x2000): Trace: sh[0x14bd850], connected[1], ops[(nil)], ldap[0x1424260], destructor_lock[0], release_memory[0] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_connection_callback] (0x4000): Successfully removed connection callback. (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1415970/0x1416430 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): The symlink points to [/var/lib/sss/pipes/private/sbus-dp_default.18702] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): The path including our pid is [/var/lib/sss/pipes/private/sbus-dp_default.18702] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): Removed the symlink (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_client_destructor] (0x0400): Removed PAM client (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_client_destructor] (0x0400): Removed NSS client
I can duplicate this by manually taking down the Infiniband link:
[root@node24 ~]# service sssd status sssd (pid 9132) is running... [root@node24 ~]# ifdown ib0 [root@node24 ~]# service sssd status sssd dead but pid file exists
I have also noticed that sssd will not start on boot. As I know that Infiniband tends to flutter a little bit before the link comes up, I’m thinking this is probably the same cause.
Can anyone explain this behavior and tell me what I might do to prevent it?
-- ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* || \UTGERS |---------------------*O*--------------------- ||_// Biomedical | Ryan Novosielski - Senior Technologist || \ and Health | novosirj@rutgers.edu - 973/972.0922 (2x0922) || \ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark `'
On Tue, Nov 03, 2015 at 04:15:57PM -0500, Novosielski, Ryan wrote:
Over time, I’ve been having seemingly random sssd quits that I’ve not been able to figure out. Today, I finally traced it to fluctuations on my Infiniband fabric:
sssd.log
(Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16 (Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface idx 4 (ib0) flags 0x1003 (broadcast,multicast,up) (Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16 (Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface idx 4 (ib0) flags 0x11043 (broadcast,multicast,up,running,lower)
These messages should just tell sssd to reset its online/offline status. It's possible we have a bug there, but this code should be pretty stable, there were no changes there recently. Maybe the services don't handle the resetOffline signal well, but that's just my speculation.
There's no way to disable the netlink integration except for build changes.
This exactly corresponds to the time in /var/log/messages for the unexplained shutdown:
2015-11-03T13:17:59-05:00 node75 sssd[pam]: Shutting down 2015-11-03T13:17:59-05:00 node75 sssd[be[default]]: Shutting down 2015-11-03T13:17:59-05:00 node75 sssd[nss]: Shutting down
Here is sssd_default.log for good measure:
(Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1414770/0x14133d0 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1414770/0x13fef90 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_ptask_destructor] (0x0400): Terminating periodic task [Cleanup of default] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sdap_handle_release] (0x2000): Trace: sh[0x14bd850], connected[1], ops[(nil)], ldap[0x1424260], destructor_lock[0], release_memory[0] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_connection_callback] (0x4000): Successfully removed connection callback. (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [sbus_remove_watch] (0x2000): 0x1415970/0x1416430 (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): The symlink points to [/var/lib/sss/pipes/private/sbus-dp_default.18702] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): The path including our pid is [/var/lib/sss/pipes/private/sbus-dp_default.18702] (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [remove_socket_symlink] (0x4000): Removed the symlink (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_client_destructor] (0x0400): Removed PAM client (Tue Nov 3 13:17:59 2015) [sssd[be[default]]] [be_client_destructor] (0x0400): Removed NSS client
Is there anything else in the logs? Can you look for messages just before the "server_setup" line? (The server_setup function is the first one we print after startup)
What about the sssd.log itself?
I can duplicate this by manually taking down the Infiniband link:
[root@node24 ~]# service sssd status sssd (pid 9132) is running... [root@node24 ~]# ifdown ib0 [root@node24 ~]# service sssd status sssd dead but pid file exists
What SSSD version are you running on what OS?
If it's RHEL/Centos or some of its derivatives, can you enable abrt to see if there's any crash? Are there any sssd-related messages in syslog?
On (04/11/15 10:10), Jakub Hrozek wrote:
On Tue, Nov 03, 2015 at 04:15:57PM -0500, Novosielski, Ryan wrote:
Over time, I’ve been having seemingly random sssd quits that I’ve not been able to figure out. Today, I finally traced it to fluctuations on my Infiniband fabric:
sssd.log
(Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16 (Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface idx 4 (ib0) flags 0x1003 (broadcast,multicast,up) (Tue Nov 3 13:17:59 2015) [sssd] [message_type] (0x0200): netlink Message type: 16 (Tue Nov 3 13:17:59 2015) [sssd] [link_msg_handler] (0x1000): netlink link message: iface idx 4 (ib0) flags 0x11043 (broadcast,multicast,up,running,lower)
These messages should just tell sssd to reset its online/offline status. It's possible we have a bug there, but this code should be pretty stable, there were no changes there recently. Maybe the services don't handle the resetOffline signal well, but that's just my speculation.
There's no way to disable the netlink integration except for build changes.
Today, we merged the patch for disablink netlink integration. https://git.fedorahosted.org/cgit/sssd.git/commit/?id=7c475c3a02329f477f6245... https://fedorahosted.org/sssd/ticket/2860
It will be part of sssd 1.14.1 which should be released in few days.
LS
Lukas Slebodnik wrote:
Today, we merged the patch for disablink netlink integration. https://git.fedorahosted.org/cgit/sssd.git/commit/?id=7c475c3a02329f477f6245... https://fedorahosted.org/sssd/ticket/2860
Good.
Do I understand the patch correctly that it's a command-line option to be used in init scripts? If yes, why not a configuration option in sssd.conf?
Ciao, Michael.
On 18 Aug 2016, at 17:16, Michael Ströder michael@stroeder.com wrote:
Lukas Slebodnik wrote:
Today, we merged the patch for disablink netlink integration. https://git.fedorahosted.org/cgit/sssd.git/commit/?id=7c475c3a02329f477f6245... https://fedorahosted.org/sssd/ticket/2860
Good.
Do I understand the patch correctly that it's a command-line option to be used in init scripts? If yes, why not a configuration option in sssd.conf?
No particular reason, when I talked with Justin about implementing this it frankly didn't occur to me that the option could be used outside tests (where we want to simulate offline lookups w/o interference from the system). I totally forgot about this thread.
Do you also think this option would be useful in general? (I'm wary of adding more options, imo we already have too many..)
Jakub Hrozek wrote:
On 18 Aug 2016, at 17:16, Michael Ströder michael@stroeder.com wrote:
Lukas Slebodnik wrote:
Today, we merged the patch for disablink netlink integration. https://git.fedorahosted.org/cgit/sssd.git/commit/?id=7c475c3a02329f477f6245... https://fedorahosted.org/sssd/ticket/2860
Good.
Do I understand the patch correctly that it's a command-line option to be used in init scripts? If yes, why not a configuration option in sssd.conf?
No particular reason, when I talked with Justin about implementing this it frankly didn't occur to me that the option could be used outside tests (where we want to simulate offline lookups w/o interference from the system). I totally forgot about this thread.
Do you also think this option would be useful in general? (I'm wary of adding more options, imo we already have too many..)
My customer currently builds minimal custom Debian packages disabling netlink support because it does not make sense in a data center.
But in general using OS-provided packages would be preferred. And not having to muck with starts scripts provided by this packages would be appreciated too.
=> IMHO such an option in sssd.conf for disabling netlink support completely would be useful.
Ciao, Michael.
On Thu, Aug 18, 2016 at 10:52:09PM +0200, Michael Ströder wrote:
Jakub Hrozek wrote:
On 18 Aug 2016, at 17:16, Michael Ströder michael@stroeder.com wrote:
Lukas Slebodnik wrote:
Today, we merged the patch for disablink netlink integration. https://git.fedorahosted.org/cgit/sssd.git/commit/?id=7c475c3a02329f477f6245... https://fedorahosted.org/sssd/ticket/2860
Good.
Do I understand the patch correctly that it's a command-line option to be used in init scripts? If yes, why not a configuration option in sssd.conf?
No particular reason, when I talked with Justin about implementing this it frankly didn't occur to me that the option could be used outside tests (where we want to simulate offline lookups w/o interference from the system). I totally forgot about this thread.
Do you also think this option would be useful in general? (I'm wary of adding more options, imo we already have too many..)
My customer currently builds minimal custom Debian packages disabling netlink support because it does not make sense in a data center.
But in general using OS-provided packages would be preferred. And not having to muck with starts scripts provided by this packages would be appreciated too.
=> IMHO such an option in sssd.conf for disabling netlink support completely would be useful.
OK, I filed: https://fedorahosted.org/sssd/ticket/3142
On (18/08/16 22:52), Michael Ströder wrote:
Jakub Hrozek wrote:
On 18 Aug 2016, at 17:16, Michael Ströder michael@stroeder.com wrote:
Lukas Slebodnik wrote:
Today, we merged the patch for disablink netlink integration. https://git.fedorahosted.org/cgit/sssd.git/commit/?id=7c475c3a02329f477f6245... https://fedorahosted.org/sssd/ticket/2860
Good.
Do I understand the patch correctly that it's a command-line option to be used in init scripts? If yes, why not a configuration option in sssd.conf?
No particular reason, when I talked with Justin about implementing this it frankly didn't occur to me that the option could be used outside tests (where we want to simulate offline lookups w/o interference from the system). I totally forgot about this thread.
Do you also think this option would be useful in general? (I'm wary of adding more options, imo we already have too many..)
My customer currently builds minimal custom Debian packages disabling netlink support because it does not make sense in a data center.
But in general using OS-provided packages would be preferred. And not having to muck with starts scripts provided by this packages would be appreciated too.
=> IMHO such an option in sssd.conf for disabling netlink support completely would be useful.
IMHO command line option is enough. Systemd support merging of configuration file.
[root@host ~]# cat /etc/systemd/system/sssd.service.d/netlink.conf [Service] # Uncomment *both* of the following lines to ignore Netlink # changes when makin decisions about resetting online and offline # operational status. You will need to run 'systemctl daemon-reload' # and then restart the SSSD service for this to take effect #ExecStart= #ExecStart=/usr/sbin/sssd -D -f --disable-netlink
LS
sssd-users@lists.fedorahosted.org
-
Jakub Hrozek -
Jakub Hrozek -
Lukas Slebodnik -
Michael Ströder -
Ryan Novosielski