Hi Lukas,
On 23 March 2016 at 15:58, Lukas Slebodnik <lslebodn(a)redhat.com> wrote:
On (23/03/16 15:49), Patrick Coleman wrote:
> $ grep network_status_change_cb sssd.log | grep '06:59:12' | wc -l
> 1245
I believe the root cause of our problem is that sssd is receiving more
than a thousand netlink notifications a second - because of IPv6 route
updates - per above. While this might be something we can reduce on
our end, it would be great if there was some way to unsubscribe sssd
from receiving these.
Any ideas here appreciated - eg. if netlink is only used to get a more
rapid notification of network changes, could we remove the call to
setup_netlink in src/monitor/monitor.c in a local build? Does sssd
rely on netlink for anything else?
> (Wed Mar 23 06:59:07 2016) [sssd] [route_msg_debug_print]
(0x1000): route idx 209591 flags 0X200 family 10 addr
fd0a:9b09:1f7:0:218:aff:fe33:3b66/128
> (Wed Mar 23 06:59:07 2016) [sssd] [network_status_change_cb] (0x2000): A networking
status change detected signaling providers to reset offline status
If I interpret this correctly, every single netlink notification seems
to be causing sssd to send a message over dbus?
The real error is in sssd_$domain.log
neither sssd.log nor sssd_nss.log will help you.
I see nothing wrong in the domain logs. I've included samples of
normal operation before the failure this morning in the domain log
below in any case. If there's anything further I can provide let me
know.
Cheers,
Patrick
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_initgr_rfc2307_next_base] (0x0400): Searching for groups with
base [dc=meraki,dc=com]
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(memberuid=blinken)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=meraki,dc=com].
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no
errmsg set
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_groups_next_base] (0x0400): Searching for groups with base
[dc=meraki,dc=com]
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(gidNumber=1111)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=meraki,dc=com].
(Wed Mar 23 06:54:04 2016) [sssd[be[meraki]]]
[sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no
errmsg set
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_groups_process] (0x0400): Search for groups, returned 1
results.
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_attrs_get_sid_str] (0x0080): No [objectSID] attribute while
id-mapping. [0][Success]
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_primary_name] (0x0400): Processing object blinken
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_group]
(0x0400): Processing group blinken
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_group]
(0x0400): Original USN value is not available for [blinken].
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_process_ghost_members] (0x0400): Group has 0 members
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_group]
(0x0400): Storing info for group blinken
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sysdb_search_group_by_name] (0x0400): No such entry
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sysdb_search_group_by_gid] (0x0400): No such entry
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [acctinfo_callback]
(0x0100): Request processed. Returned 0,0,Success
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [be_get_account_info]
(0x0100): Got request for [4099][1][name=blinken]
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [be_req_set_domain]
(0x0400): Changing request domain from [ourdomain] to [ourdomain]
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_initgr_next_base] (0x0400): Searching for users with base
[dc=meraki,dc=com]
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(uid=blinken)(objectclass=posixAccount)(&(uidNumber=*)(!(uidNumber=0))))][dc=meraki,dc=com].
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no
errmsg set
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user]
(0x0400): Save user
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_attrs_get_sid_str] (0x0080): No [objectSID] attribute while
id-mapping. [0][Success]
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sdap_get_primary_name] (0x0400): Processing object blinken
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user]
(0x0400): Processing user blinken
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user]
(0x0400): Original memberOf is not available for [blinken].
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user]
(0x0400): Original USN value is not available for [blinken].
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user]
(0x0400): User principal is not available for [blinken].
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]] [sdap_save_user]
(0x0400): Storing info for user blinken
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sysdb_search_user_by_name] (0x0400): No such entry
(Wed Mar 23 06:54:04 2016) [sssd[be[ourdomain]]]
[sysdb_search_user_by_uid] (0x0400): No such entry
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_initgr_rfc2307_next_base] (0x0400): Searching for groups with
base [dc=meraki,dc=com]
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(memberuid=blinken)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=meraki,dc=com].
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no
errmsg set
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_get_groups_next_base] (0x0400): Searching for groups with base
[dc=meraki,dc=com]
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_step] (0x0400): calling ldap_search_ext with
[(&(gidNumber=1111)(objectClass=posixGroup)(cn=*)(&(gidNumber=*)(!(gidNumber=0))))][dc=meraki,dc=com].
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_get_generic_ext_done] (0x0400): Search result: Success(0), no
errmsg set
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_get_groups_process] (0x0400): Search for groups, returned 1
results.
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_attrs_get_sid_str] (0x0080): No [objectSID] attribute while
id-mapping. [0][Success]
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_get_primary_name] (0x0400): Processing object blinken
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_save_group]
(0x0400): Processing group blinken
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_save_group]
(0x0400): Original USN value is not available for [blinken].
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sdap_process_ghost_members] (0x0400): Group has 0 members
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]] [sdap_save_group]
(0x0400): Storing info for group blinken
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sysdb_search_group_by_name] (0x0400): No such entry
(Wed Mar 23 06:54:05 2016) [sssd[be[ourdomain]]]
[sysdb_search_group_by_gid] (0x0400): No such entry