[SSSD] sssd pam backend crashing once in a while.
Steve Traylen
steve.traylen at cern.ch
Tue May 21 13:55:00 UTC 2013
On May 21, 2013, at 12:24 PM, Jakub Hrozek wrote:
> On Tue, May 21, 2013 at 09:58:06AM +0000, Steve Traylen wrote:
>>
>> Hi,
>>
>> We are experienced trouble in our interactive login service where sssd in crashing every few hours in most of the nodes.
>> We run on Scientific Linux CERN SLC release 6.4 (Carbon) (which is RH 6.4 based), more precisely:
>>
>
> Hi,
> I'm sorry for the trouble.
>
:-) No need to apologize.
>> sssd-1.9.2-82.7.el6_4.x86_64
>> sssd-client-1.9.2-82.7.el6_4.x86_64
>>
>
> That's pretty much the latest stable release.
>
>> We have some logs
>>
>> (Mon May 20 14:31:27 2013) [sssd[pam]] [sss_dp_init] (0x0010): Failed to connect to monitor services.
>> (Mon May 20 14:31:27 2013) [sssd[pam]] [sss_process_init] (0x0010): fatal error setting up backend connector
>>
>> which is the restart of the backed failing?
>>
>>
>
> It's actually the PAM process not succeeding in reconnecting to the UNIX
> pipe connecting it with the monitor process that watches over all the
> SSSD worker processes.
>
>>
>> Here is our sssd.conf
>>
> [snip]
>
> The config file looks good to me.
>
>>
>> and a backtrace below. I can provide a core file.
>>
>
> Providing the core file would be great if you don't mind. However, keep
> in mind that the core file might contain confidential information, in
> some cases even the passwords.
>
Hi Jakub,
I'll make the core files to available to you offline. In fact we have a selection of core files:
1) sssd_be_core.14772
#0 dbus_watch_handle (watch=0x90, flags=2) at dbus-watch.c:650
#1 0x000000000046edcc in sbus_watch_handler (ev=<value optimized out>, fde=<value optimized out>,
flags=<value optimized out>, data=<value optimized out>) at src/sbus/sssd_dbus_common.c:93
#2 0x00007fcb8f3e23ff in epoll_event_loop (ev=<value optimized out>, location=<value optimized out>)
at ../tevent_standard.c:328
#3 std_event_loop_once (ev=<value optimized out>, location=<value optimized out>) at ../tevent_standard.c:567
#4 0x00007fcb8f3de8f0 in _tevent_loop_once (ev=0x176a590, location=0x47c713 "src/util/server.c:601")
at ../tevent.c:507
#5 0x00007fcb8f3de95b in tevent_common_loop_wait (ev=0x176a590, location=0x47c713 "src/util/server.c:601")
at ../tevent.c:608
#6 0x0000000000451973 in server_loop (main_ctx=0x176b6a0) at src/util/server.c:601
#7 0x000000000041a066 in main (argc=<value optimized out>, argv=<value optimized out>)
at src/providers/data_provider_be.c:2732
2) sssd_nss_core.32675
#0 dbus_watch_handle (watch=0x90, flags=2) at dbus-watch.c:650
#1 0x000000000047227c in sbus_watch_handler (ev=<value optimized out>, fde=<value optimized out>,
flags=<value optimized out>, data=<value optimized out>) at src/sbus/sssd_dbus_common.c:93
#2 0x00007fbd5c5623ff in epoll_event_loop (ev=<value optimized out>, location=<value optimized out>)
at ../tevent_standard.c:328
#3 std_event_loop_once (ev=<value optimized out>, location=<value optimized out>) at ../tevent_standard.c:567
#4 0x00007fbd5c55e8f0 in _tevent_loop_once (ev=0x89e3b0, location=0x4816a3 "src/util/server.c:601")
at ../tevent.c:507
#5 0x00007fbd5c55e95b in tevent_common_loop_wait (ev=0x89e3b0, location=0x4816a3 "src/util/server.c:601")
at ../tevent.c:608
#6 0x000000000045a1b3 in server_loop (main_ctx=0x89f530) at src/util/server.c:601
#7 0x00000000004090a0 in main (argc=<value optimized out>, argv=<value optimized out>)
3) sssd_pam_core.18894
#0 0x00007f297c2778a5 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007f297c279085 in abort () at abort.c:92
#2 0x00007f297f41cc3c in talloc_abort (reason=0x7f297f422378 "Bad talloc magic value - unknown value")
at ../talloc.c:317
#3 0x00007f297f41cdf1 in talloc_abort_unknown_value (ptr=<value optimized out>) at ../talloc.c:341
#4 talloc_chunk_from_ptr (ptr=<value optimized out>) at ../talloc.c:360
#5 talloc_get_name (ptr=<value optimized out>) at ../talloc.c:1153
#6 0x00007f297f41ce1e in talloc_check_name (ptr=0x1fe4c90, name=0x461f5d "struct pam_auth_req")
at ../talloc.c:1172
#7 0x0000000000410e0a in pam_dp_process_reply (pending=0x1fdd250, ptr=<value optimized out>)
at src/responder/pam/pamsrv_dp.c:42
#8 0x00007f297edb161a in complete_pending_call_and_unlock (connection=0x1fb9780, pending=0x1fdd250,
message=<value optimized out>) at dbus-connection.c:2234
#9 0x00007f297edb386f in dbus_connection_dispatch (connection=0x1fb9780) at dbus-connection.c:4397
#10 0x000000000045425e in sbus_dispatch (ev=0x1fad3d0, te=<value optimized out>, tv=...,
data=<value optimized out>) at src/sbus/sssd_dbus_connection.c:104
#11 0x00007f297f62bbd9 in tevent_common_loop_timer_delay (ev=0x1fad3d0) at ../tevent_timed.c:254
#12 0x00007f297f62b2ab in std_event_loop_once (ev=<value optimized out>, location=<value optimized out>)
at ../tevent_standard.c:560
#13 0x00007f297f6278f0 in _tevent_loop_once (ev=0x1fad3d0, location=0x46d063 "src/util/server.c:601")
at ../tevent.c:507
#14 0x00007f297f62795b in tevent_common_loop_wait (ev=0x1fad3d0, location=0x46d063 "src/util/server.c:601")
at ../tevent.c:608
#15 0x0000000000455bb3 in server_loop (main_ctx=0x1fae550) at src/util/server.c:601
#16 0x0000000000409b32 in main (argc=<value optimized out>, argv=<value optimized out>)
at src/responder/pam/pamsrv.c:260
>> Cheers; Steve
>>
>> # ls -ltd /core.18894
>> -rw-------. 1 root root 1253376 May 14 20:10 /core.18894
>> # file /core.18894
>> /core.18894: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/libexec/sssd/sssd_pam --debug-to-files'
>> #
>>
>> .....does this ring a bell?
>
> No, sorry. This looks like some kind of a new bug to me. Can you install
> sssd-debuginfo (debuginfo-install sssd should be all you need) and
> generate the backtrace again?
>
> When does the crash happen, can you see any pattern in SSSD usage? I'd
> like to reproduce the crash in-house if possible.
I don't have a pattern at the moment. .. We were using sssd happily on our internal test clusters
with a limited number of users. .. When we switched a user facing (14,000 users) cluster we started
to have problems. I suspect we have more crashes when particular machines are under load and
we certainly have less (*) sssd crashes since we fixed another piece of software that was going bananas CPU
wise.
Will try and find a correlation.
(*) Of 60 (hopefully identical) nodes around 10 have had an sssd crash in the last couple of days.
>>
> _______________________________________________
> sssd-devel mailing list
> sssd-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/sssd-devel
More information about the sssd-devel
mailing list