[libqb] qb_ipcs_connection_unref() errors while using QB_IPC_SOCKET

Andrei Belov defanator at gmail.com
Wed Apr 10 09:40:43 UTC 2013


On Apr 10, 2013, at 10:46 , Andrei Belov <defanator at gmail.com> wrote:

> 
> On Apr 10, 2013, at 10:00 , Andrei Belov <defanator at gmail.com> wrote:
> 
>> Angus,
>> 
>> On Apr 10, 2013, at 9:24 , Angus Salkeld <asalkeld at redhat.com> wrote:
>> 
>>> On 10/04/13 08:52 +0400, Andrei Belov wrote:
>>>> Hello,
>>>> 
>>>> I'm facing the issue with qp_ipcs_connection_unref() errors while
>>>> using libqb (0.14.4) with pacemaker (1.1.8) under SunOS
>>>> with QB_IPC_SOCKET.
>>>> 
>>>> Pacemaker's daemons are being aborted on shutdown with the following in logs:
>>>> 
>>>> pr 10 04:05:06 [47131]      attrd:    error: qb_ipcs_connection_unref:         ref:0 state:3 (47131-47133-11)
>>>> Apr 10 04:05:13 [47130]       lrmd:    error: qb_ipcs_connection_unref:         ref:0 state:3 (47130-47133-9)
>>>> Apr 10 04:05:20 [47129] stonith-ng:    error: qb_ipcs_connection_unref:         ref:0 state:3 (47129-47133-13)
>>>> Apr 10 04:05:27 [47128]        cib:    error: qb_ipcs_connection_unref:         ref:0 state:3 (47128-47131-19)
>>>> 
>>>> What does it mean?
>>>> 
>>>> 
>>>> Backtraces:
>>>> 
>>>> Core was generated by `/opt/local/libexec/pacemaker/attrd'.
>>>> Program terminated with signal 6, Aborted.
>>>> #0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
>>>> (gdb) bt
>>>> #0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
>>>> #1  0xfffffd7fff0d4ddd in thr_kill () from /lib/64/libc.so.1
>>>> #2  0xfffffd7fff06a971 in raise () from /lib/64/libc.so.1
>>>> #3  0xfffffd7fff0400a1 in abort () from /lib/64/libc.so.1
>>>> #4  0xfffffd7fff0403f5 in _assert () from /lib/64/libc.so.1
>>>> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
>>>> #6  0x00000000004044f9 in main ()
>>> 
>>> Hi
>>> 
>>> The connection is reference counted, and it is getting dereferenced
>>> one too many times. This is due to differences in the behaviour of
>>> sockets on solaris and linux. Basically the code path for solaris
>>> shutdown is not well tested by me.
>>> 
>>> (patches welcome)
>> 
>> Thanks, I'll try to examine this deeper.
>> 
>> I see "tests" subdirectory in libqb - what is the right way to run those?
> 
> Just tried "distcheck" target, IPC test failed with the following:
> 
> ../../tests/check_ipc.c:704:E:ipc_event_on_created_us:test_ipc_event_on_created_us:0: (after this point) Test timeout expired
> ../../tests/check_ipc.c:757:E:ipc_disconnect_after_created_us:test_ipc_disconnect_after_created_us:0: (after this point) Test timeout expired
> 
> Full output is here:
> http://defan.pp.ru/libqb-sunos-distcheck.txt
> 
> 
> Still trying to understand what happens.

Here's the full detailed output of ipc-test only:
http://defan.pp.ru/libqb-sunos-ipc-test.txt.gz

It ended up with:

[..]
lib/check_ipc.c|787| ENTERING test_ipc_disconnect_after_created_us()
lib/ipc_setup.c|372| server name: test_ipc_disconnect_after_created_us
lib/loop_poll.c|357| grown poll array to 2 for FD 6
lib/ipc_setup.c|476| IPC credentials authenticated (29647-29646-7)
lib/ipc_socket.c|527| connecting to client (29647-29646-7)
lib/ipc_setup.c|524| Error in connection setup (29647-29646-7): Not owner (1)
lib/ipcs.c|555| qb_ipcs_disconnect(29647-29646-7) state:0
lib/ipc_setup.c|476| IPC credentials authenticated (29647-29646-8)
lib/ipc_socket.c|527| connecting to client (29647-29646-8)
lib/ipc_setup.c|524| Error in connection setup (29647-29646-8): Not owner (1)
lib/ipcs.c|555| qb_ipcs_disconnect(29647-29646-8) state:0
lib/ipc_setup.c|476| IPC credentials authenticated (29647-29646-9)
lib/ipc_socket.c|527| connecting to client (29647-29646-9)
lib/ipc_setup.c|524| Error in connection setup (29647-29646-9): Not owner (1)
lib/ipcs.c|555| qb_ipcs_disconnect(29647-29646-9) state:0
77%: Checks: 9, Failures: 0, Errors: 2
../../tests/check_ipc.c:840:P:ipc_server_fail_soc:test_ipc_server_fail_soc:0: Passed
../../tests/check_ipc.c:304:P:ipc_txrx_us_block:test_ipc_txrx_us_block:0: Passed
../../tests/check_ipc.c:304:P:ipc_txrx_us_tmo:test_ipc_txrx_us_tmo:0: Passed
../../tests/check_ipc.c:383:P:ipc_fc_us:test_ipc_fc_us:0: Passed
../../tests/check_ipc.c:424:P:ipc_exit_us:test_ipc_exit_us:0: Passed
../../tests/check_ipc.c:304:P:ipc_dispatch_us:test_ipc_disp_us:0: Passed
../../tests/check_ipc.c:654:P:ipc_bulk_events_us:test_ipc_bulk_events_us:0: Passed
../../tests/check_ipc.c:704:E:ipc_event_on_created_us:test_ipc_event_on_created_us:0: (after this point) Test timeout expired
../../tests/check_ipc.c:757:E:ipc_disconnect_after_created_us:test_ipc_disconnect_after_created_us:0: (after this point) Test timeout expired
FAIL: ipc.test
[..]


> 
> 
>>> -Angus
>>> 
>>>> 
>>>> Core was generated by `/opt/local/libexec/pacemaker/cib'.
>>>> Program terminated with signal 6, Aborted.
>>>> #0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
>>>> (gdb) bt
>>>> #0  0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
>>>> #1  0xfffffd7fff0e4ddd in thr_kill () from /lib/64/libc.so.1
>>>> #2  0xfffffd7fff07a971 in raise () from /lib/64/libc.so.1
>>>> #3  0xfffffd7fff0500a1 in abort () from /lib/64/libc.so.1
>>>> #4  0xfffffd7fff0503f5 in _assert () from /lib/64/libc.so.1
>>>> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
>>>> #6  0x0000000000410438 in cib_shutdown ()
>>>> #7  0xfffffd7fbfc5533f in crm_signal_dispatch (source=0x49be80, callback=<optimized out>, userdata=<optimized out>)
>>>> at mainloop.c:203
>>>> #8  0xfffffd7fc555f9e0 in g_main_context_dispatch () from /opt/local/lib/libglib-2.0.so.0
>>>> #9  0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from /opt/local/lib/libglib-2.0.so.0
>>>> #10 0xfffffd7fc5560152 in g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
>>>> #11 0x0000000000411056 in cib_init ()
>>>> #12 0x000000000041163e in main ()
>>>> 
>>>> Core was generated by `/opt/local/libexec/pacemaker/lrmd'.
>>>> Program terminated with signal 6, Aborted.
>>>> #0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
>>>> (gdb) bt
>>>> #0  0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
>>>> #1  0xfffffd7fff0d4ddd in thr_kill () from /lib/64/libc.so.1
>>>> #2  0xfffffd7fff06a971 in raise () from /lib/64/libc.so.1
>>>> #3  0xfffffd7fff0400a1 in abort () from /lib/64/libc.so.1
>>>> #4  0xfffffd7fff0403f5 in _assert () from /lib/64/libc.so.1
>>>> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
>>>> #6  0xfffffd7fc02128a4 in qb_ipcs_disconnect () from /opt/local/lib/libqb.so.0
>>>> #7  0xfffffd7fc0212995 in qb_ipcs_unref () from /opt/local/lib/libqb.so.0
>>>> #8  0xfffffd7fc02129c7 in qb_ipcs_destroy () from /opt/local/lib/libqb.so.0
>>>> #9  0xfffffd7fbfc55a3f in mainloop_del_ipc_server (server=<optimized out>) at mainloop.c:517
>>>> #10 0x00000000004041cd in lrmd_shutdown ()
>>>> #11 0xfffffd7fbfc5533f in crm_signal_dispatch (source=0x48ad40, callback=<optimized out>, userdata=<optimized out>)
>>>> at mainloop.c:203
>>>> #12 0xfffffd7fc555f9e0 in g_main_context_dispatch () from /opt/local/lib/libglib-2.0.so.0
>>>> #13 0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from /opt/local/lib/libglib-2.0.so.0
>>>> #14 0xfffffd7fc5560152 in g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
>>>> #15 0x00000000004045e3 in main ()
>>>> 
>>>> Core was generated by `/opt/local/libexec/pacemaker/stonithd'.
>>>> Program terminated with signal 6, Aborted.
>>>> #0  0xfffffd7fff11061a in _lwp_kill () from /lib/64/libc.so.1
>>>> (gdb) bt
>>>> #0  0xfffffd7fff11061a in _lwp_kill () from /lib/64/libc.so.1
>>>> #1  0xfffffd7fff104ddd in thr_kill () from /lib/64/libc.so.1
>>>> #2  0xfffffd7fff09a971 in raise () from /lib/64/libc.so.1
>>>> #3  0xfffffd7fff0700a1 in abort () from /lib/64/libc.so.1
>>>> #4  0xfffffd7fff0703f5 in _assert () from /lib/64/libc.so.1
>>>> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
>>>> #6  0xfffffd7fc02128a4 in qb_ipcs_disconnect () from /opt/local/lib/libqb.so.0
>>>> #7  0xfffffd7fc0212995 in qb_ipcs_unref () from /opt/local/lib/libqb.so.0
>>>> #8  0xfffffd7fc02129c7 in qb_ipcs_destroy () from /opt/local/lib/libqb.so.0
>>>> #9  0x0000000000405e60 in ?? ()
>>>> #10 0x0000000000407d28 in main ()
>>>> 
>>>> 
>>>> PS:
>>>> while reproducing the above, just caught the same with corosync (2.3.0):
>>>> 
>>>> Core was generated by `corosync'.
>>>> Program terminated with signal 6, Aborted.
>>>> #0  0xfffffd7fff1e061a in _lwp_kill () from /lib/64/libc.so.1
>>>> (gdb) bt
>>>> #0  0xfffffd7fff1e061a in _lwp_kill () from /lib/64/libc.so.1
>>>> #1  0xfffffd7fff1d4ddd in thr_kill () from /lib/64/libc.so.1
>>>> #2  0xfffffd7fff16a971 in raise () from /lib/64/libc.so.1
>>>> #3  0xfffffd7fff1400a1 in abort () from /lib/64/libc.so.1
>>>> #4  0xfffffd7fff1403f5 in _assert () from /lib/64/libc.so.1
>>>> #5  0xfffffd7fc021274e in qb_ipcs_connection_unref () from /opt/local/lib/libqb.so.0
>>>> #6  0xfffffd7fc02128a4 in qb_ipcs_disconnect () from /opt/local/lib/libqb.so.0
>>>> #7  0xfffffd7fc0212995 in qb_ipcs_unref () from /opt/local/lib/libqb.so.0
>>>> #8  0xfffffd7fc02129c7 in qb_ipcs_destroy () from /opt/local/lib/libqb.so.0
>>>> #9  0x00000000004264e2 in cs_ipcs_service_destroy ()
>>>> #10 0x00000000004270e0 in ?? ()
>>>> #11 0xfffffd7fc0210ff4 in job_dispatch () from /opt/local/lib/libqb.so.0
>>>> #12 0xfffffd7fc020fcb4 in qb_loop_run () from /opt/local/lib/libqb.so.0
>>>> #13 0x000000000042b551 in main ()
>>>> 
>>>> so it looks like libqb related issue for me.
>>>> 
>>>> 
>>>> Any help would be greatly appreciated!
>>>> 
>>>> 
>>>> Best regards,
>>>> Andrei
>>>> _______________________________________________
>>>> quarterback-devel mailing list
>>>> quarterback-devel at lists.fedorahosted.org
>>>> https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel
>>> _______________________________________________
>>> quarterback-devel mailing list
>>> quarterback-devel at lists.fedorahosted.org
>>> https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel
>> 
> 



More information about the quarterback-devel mailing list