[SSSD] Question about pam_auth_req and crash from BZ972699
Jakub Hrozek
jhrozek at redhat.com
Wed Sep 4 10:03:32 UTC 2013
On Wed, Sep 04, 2013 at 11:23:54AM +0200, Sumit Bose wrote:
> On Wed, Sep 04, 2013 at 08:50:47AM +0200, Jakub Hrozek wrote:
> > On Tue, Aug 20, 2013 at 10:32:35PM +0200, Lukas Slebodnik wrote:
> > > On (20/08/13 14:17), Jakub Hrozek wrote:
> > > >On Tue, Aug 20, 2013 at 02:00:35PM +0200, Lukas Slebodnik wrote:
> > > >> ehlo,
> > > >>
> > > >> I have a question about struct pam_auth_req and crash from BZ972699
> > > >> I thought It is a duplicate of upstream ticket
> > > >> https://fedorahosted.org/sssd/ticket/2018 (use after free)
> > > >>
> > > >> But I anlysed the core dump and it doe not look like use after free.
> > > >> I also checked talloc_magic for struct "pam_auth_req" and it is ok.
> > > >>
> > > >> Here is the definition of structure.
> > > >> (gdb) ptype preq
> > > >> type = struct pam_auth_req {
> > > >> struct cli_ctx *cctx;
> > > >> struct sss_domain_info *domain;
> > > >> struct pam_data *pd;
> > > >> pam_dp_callback_t *callback;
> > > >> struct ldb_result *res;
> > > >> _Bool check_provider;
> > > >> void *data;
> > > >> } *
> > > >>
> > > >>
> > > >> And here is output from gdb
> > > >>
> > > >> (gdb) p *preq
> > > >> $1 = {cctx = 0x9f90a0, domain = 0x9a3740, pd = 0xa178f0, callback = 0,
> > > >> res = 0x0, check_provider = false, data = 0x0}
> > > >>
> > > >> #0 0x0000000000000000 in ?? ()
> > > >> #1 0x00000000004110d1 in pam_dp_process_reply (pending=0x9d3c00,
> > > >> ptr=<value optimized out>) at src/responder/pam/pamsrv_dp.c:79
> > > >> #2 0x00007f1aeb0a661a in complete_pending_call_and_unlock (
> > > >> connection=0xa1e3a0, pending=0x9d3c00, message=<value optimized out>)
> > > >> at dbus-connection.c:2234
> > > >> #3 0x00007f1aeb0a886f in dbus_connection_dispatch (connection=0xa1e3a0)
> > > >> at dbus-connection.c:4397
> > > >> #4 0x000000000045425e in sbus_dispatch (ev=0x99d380,
> > > >> te=<value optimized out>, tv=..., data=<value optimized out>)
> > > >> at src/sbus/sssd_dbus_connection.c:104
> > > >> #5 0x00007f1aeb920bd9 in tevent_common_loop_timer_delay (ev=0x99d380)
> > > >> at ../tevent_timed.c:254
> > > >>
> > > >> 77 dbus_pending_call_unref(pending);
> > > >> 78 dbus_message_unref(msg);
> > > >> 79 preq->callback(preq); <<<<<<< CHASH HERE
> > > >> 80 }
> > > >> 81
> > > >>
> > > >> sssd_pam crashed because preq->callback was NULL.
> > > >>
> > > >> I would like to describe struct pam_auth_req more deeply.
> > > >> struct pam_auth_req is created in function pam_forwarder
> > > >> form file "src/responder/pam/pamsrv_cmd.c".
> > > >>
> > > >> Most of structure members are initialized in this function, but *calback*
> > > >> is initialized in function pam_dom_forwarder.
> > > >> 1173 if (!NEED_CHECK_PROVIDER(preq->domain->provider)) {
> > > >> 1174 preq->callback = pam_reply; <<<<HERE
> > > >> 1175 ret = LOCAL_pam_handler(preq);
> > > >> 1176 }
> > > >> 1177 else {
> > > >> 1178 preq->callback = pam_reply; <<<<< OR HERE
> > > >
> > > >It was probably this one, the other call is only called for
> > > >id_provider=local
> > > >
> > > preq->callback is NULL, pam_dom_forwarder could not be called.
> > > Output from gdb is in this mail few lines before.
> > >
> > > >> 1179 ret = pam_dp_send_req(preq, SSS_CLI_SOCKET_TIMEOUT/2);
> > > >> 1180 DEBUG(4, ("pam_dp_send_req returned %d\n", ret));
> > > >> 1181 }
> > > >> 1182
> > > >>
> > > >> There are 3 places where is pam_dom_forwarder called and it is called only
> > > >> in some condition (pam_check_user_search returned EOK)
> > > >> Mostly it looks like following lines:
> > > >> 852 ret = pam_check_user_search(preq);
> > > >> 853 if (ret == EOK) {
> > > >> 854 pam_dom_forwarder(preq);
> > > >> 855 }
> > > >>
> > > >>
> > > >> And my question are:
> > > >> 1. Do we need to initialize pam_auth_req->callback only in function
> > > >> pam_dom_forwarder?
> > > >
> > > >We could also move the callback to src/responder/pam/pamsrv_dp.c and
> > > >either assign it from pam_dp_send_req() or call it directly from
> > > >pam_dp_process_reply() because the way I read the code we can only ever
> > > >call pam_dp_send_req with cb == pam_reply;
> > > >
> > > >> 2. Should we check preq->callback for NULL?
> > > >
> > > >Maybe, as a sanity check, but we still should fix the real bug.
> > > >
> > > >> 3. Do you have any idea how to reproduce it?
> > > >
> > > >Can we ask the reporter to run with valgrind? Also, did we rule out
> > > >that sssd_be might be crashing or being restarted in between the requst
> > > >is send and before it finishes?
> > > >
> > > There isn't a reproducer for this bug.
> > > Reporter wrote: "Every now and then, sssd terminates"
> > >
> > > >>
> > > >> I really don't know what is right solution.
> > > >>
> > > >> Thank you very much for any ideas.
> > > >>
> > > >> LS
> >
> > Hi,
> >
> > are there any other ideas on how to proceed with this problem? I haven't
> > found any..
>
> Maybe new logs with a higher debug level would help. Form the old logs
> attached to the BZ it looks that the auth requests just needs a long
> time and the client (sshd) start a new request before the old is
> finished. This would indicate the use after free issue which does not
> match the core dump. New logs would help to find out how SSSD is getting
> into the state where the callback is NULL.
>
> bye,
> Sumit
Lukas analyzed the core once again (the reporter uploaded new corefiles
after I sent the mail to sssd-devel) and we're now reasonably sure he is
hitting a known bug where, as you said, the request is taking a long
time, the client times out and the responder accesses freed memory that
was freed as a result of client timeout destructor.
I will just build him a scratch build, maybe he can test it out. Looks
like he's running CentOS anyway, so he might now worry about support :)
Thanks for another look at the issue.
More information about the sssd-devel
mailing list