[libqb] qb_ipcs_connection_unref() errors while usingQB_IPC_SOCKET
Andrei Belov
defanator at gmail.com
Wed Apr 10 10:07:40 UTC 2013
On Apr 10, 2013, at 13:53 , Grüninger, Andreas (LGL Extern) <Andreas.Grueninger at lgl.bwl.de> wrote:
> Andrej
>
> corosync and pacemaker must be started as a non-root user.
> Be carefull when you run the tests as root and you use the same var folder for tests and the daemons.
> You have to check the permissions on your var folder afterwards.
I run tests under my unprivileged account using standard "distcheck" target.
In this case libqb uses SOCKETDIR within working directory
(e.g., /home/defan/git/libqb/libqb-0.14.4.24-b3ca/_inst/var/run).
I'm able to successfully start both corosync and pacemaker under unprivileged
user (hacluster:haclient). Test cluster from two nodes operates normally until
I'm trying to stop pacemaker/corosync.
This happens at SmartOS (Joyent cloud):
# uname -a
SunOS xxxx 5.11 joyent_20120912T055050Z i86pc i386 i86pc Solaris
Angus - could you please take a look on the output from ipc.test? I feel a bit lost here...
>
> I copied the necessary parts from my scripts.
>
> export PCMK_ipc_type=socket
> export PREFIX=/opt/ha
> export CLUSTER_USER=hacluster
> export CLUSTER_GROUP=haclient
>
> mkdir -p $PREFIX/var
> chown $CLUSTER_USER:$CLUSTER_GROUP $PREFIX/var/
>
> mkdir -p $PREFIX/etc/corosync/uidgid.d
> (
> echo "uidgid {"
> echo " uid: `id -u ${CLUSTER_USER}`"
> echo " gid: `id -g ${CLUSTER_USER}`"
> echo "}"
> ) > $PREFIX/etc/corosync/uidgid.d/uid.conf
>
> su ${CLUSTER_USER} -c ${APPPATH}${COROSYNC}
> sleep $sleep0
> su ${CLUSTER_USER} -c ${APPPATH}${PACEMAKERD} &
>
> And FYI, it works with Solaris 11.1 and OpenIndiana 151a7.
>
> The status of our cluster with 4 virtual storage appliances:
> ..................................
> crm(live)# status
> Last updated: Wed Apr 10 11:49:08 2013
> Last change: Fri Apr 5 10:28:46 2013 via cibadmin on zd-sol-s2-v61
> Stack: corosync
> Current DC: zd-sol-s1-v61 (3232251190) - partition with quorum
> Version: 1.1.8-f49aa8c
> 2 Nodes configured, unknown expected votes
> 6 Resources configured.
>
>
> Online: [ zd-sol-s1-v61 zd-sol-s2-v61 ]
>
> ClusterNotify_SNMPTraps (ocf::lgl:ClusterNotify): Started zd-sol-s1-v61
> ClusterMon_SNMPTraps (ocf::pacemaker:ClusterMon): Started zd-sol-s1-v61
> zone_zd-sol-s61 (ocf::lgl:zpool): Started zd-sol-s2-v61
> zone_zd-sol-s62 (ocf::lgl:zpool): Started zd-sol-s2-v61
> zone_zd-sol-s60 (ocf::lgl:zpool): Started zd-sol-s2-v61
> zone_zd-sol-s63 (ocf::lgl:zpool): Started zd-sol-s2-v61
> ....................................
>
> Thanks
>
> Andreas
>
> -----Ursprüngliche Nachricht-----
> Von: quarterback-devel-bounces at lists.fedorahosted.org [mailto:quarterback-devel-bounces at lists.fedorahosted.org] Im Auftrag von Andrei Belov
> Gesendet: Mittwoch, 10. April 2013 11:41
> An: lib quarterback
> Betreff: Re: [libqb] qb_ipcs_connection_unref() errors while usingQB_IPC_SOCKET
>
>
> On Apr 10, 2013, at 10:46 , Andrei Belov <defanator at gmail.com> wrote:
>
>>
>> On Apr 10, 2013, at 10:00 , Andrei Belov <defanator at gmail.com> wrote:
>>
>>> Angus,
>>>
>>> On Apr 10, 2013, at 9:24 , Angus Salkeld <asalkeld at redhat.com> wrote:
>>>
>>>> On 10/04/13 08:52 +0400, Andrei Belov wrote:
>>>>> Hello,
>>>>>
>>>>> I'm facing the issue with qp_ipcs_connection_unref() errors while
>>>>> using libqb (0.14.4) with pacemaker (1.1.8) under SunOS with
>>>>> QB_IPC_SOCKET.
>>>>>
>>>>> Pacemaker's daemons are being aborted on shutdown with the following in logs:
>>>>>
>>>>> pr 10 04:05:06 [47131] attrd: error: qb_ipcs_connection_unref: ref:0 state:3 (47131-47133-11)
>>>>> Apr 10 04:05:13 [47130] lrmd: error: qb_ipcs_connection_unref: ref:0 state:3 (47130-47133-9)
>>>>> Apr 10 04:05:20 [47129] stonith-ng: error: qb_ipcs_connection_unref: ref:0 state:3 (47129-47133-13)
>>>>> Apr 10 04:05:27 [47128] cib: error: qb_ipcs_connection_unref: ref:0 state:3 (47128-47131-19)
>>>>>
>>>>> What does it mean?
>>>>>
>>>>>
>>>>> Backtraces:
>>>>>
>>>>> Core was generated by `/opt/local/libexec/pacemaker/attrd'.
>>>>> Program terminated with signal 6, Aborted.
>>>>> #0 0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> (gdb) bt
>>>>> #0 0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> #1 0xfffffd7fff0d4ddd in thr_kill () from /lib/64/libc.so.1
>>>>> #2 0xfffffd7fff06a971 in raise () from /lib/64/libc.so.1
>>>>> #3 0xfffffd7fff0400a1 in abort () from /lib/64/libc.so.1
>>>>> #4 0xfffffd7fff0403f5 in _assert () from /lib/64/libc.so.1
>>>>> #5 0xfffffd7fc021274e in qb_ipcs_connection_unref () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #6 0x00000000004044f9 in main ()
>>>>
>>>> Hi
>>>>
>>>> The connection is reference counted, and it is getting dereferenced
>>>> one too many times. This is due to differences in the behaviour of
>>>> sockets on solaris and linux. Basically the code path for solaris
>>>> shutdown is not well tested by me.
>>>>
>>>> (patches welcome)
>>>
>>> Thanks, I'll try to examine this deeper.
>>>
>>> I see "tests" subdirectory in libqb - what is the right way to run those?
>>
>> Just tried "distcheck" target, IPC test failed with the following:
>>
>> ../../tests/check_ipc.c:704:E:ipc_event_on_created_us:test_ipc_event_o
>> n_created_us:0: (after this point) Test timeout expired
>> ../../tests/check_ipc.c:757:E:ipc_disconnect_after_created_us:test_ipc
>> _disconnect_after_created_us:0: (after this point) Test timeout
>> expired
>>
>> Full output is here:
>> http://defan.pp.ru/libqb-sunos-distcheck.txt
>>
>>
>> Still trying to understand what happens.
>
> Here's the full detailed output of ipc-test only:
> http://defan.pp.ru/libqb-sunos-ipc-test.txt.gz
>
> It ended up with:
>
> [..]
> lib/check_ipc.c|787| ENTERING test_ipc_disconnect_after_created_us()
> lib/ipc_setup.c|372| server name: test_ipc_disconnect_after_created_us
> lib/loop_poll.c|357| grown poll array to 2 for FD 6 lib/ipc_setup.c|476| IPC credentials authenticated (29647-29646-7) lib/ipc_socket.c|527| connecting to client (29647-29646-7) lib/ipc_setup.c|524| Error in connection setup (29647-29646-7): Not owner (1) lib/ipcs.c|555| qb_ipcs_disconnect(29647-29646-7) state:0 lib/ipc_setup.c|476| IPC credentials authenticated (29647-29646-8) lib/ipc_socket.c|527| connecting to client (29647-29646-8) lib/ipc_setup.c|524| Error in connection setup (29647-29646-8): Not owner (1) lib/ipcs.c|555| qb_ipcs_disconnect(29647-29646-8) state:0 lib/ipc_setup.c|476| IPC credentials authenticated (29647-29646-9) lib/ipc_socket.c|527| connecting to client (29647-29646-9) lib/ipc_setup.c|524| Error in connection setup (29647-29646-9): Not owner (1) lib/ipcs.c|555| qb_ipcs_disconnect(29647-29646-9) state:0
> 77%: Checks: 9, Failures: 0, Errors: 2
> ../../tests/check_ipc.c:840:P:ipc_server_fail_soc:test_ipc_server_fail_soc:0: Passed
> ../../tests/check_ipc.c:304:P:ipc_txrx_us_block:test_ipc_txrx_us_block:0: Passed
> ../../tests/check_ipc.c:304:P:ipc_txrx_us_tmo:test_ipc_txrx_us_tmo:0: Passed
> ../../tests/check_ipc.c:383:P:ipc_fc_us:test_ipc_fc_us:0: Passed
> ../../tests/check_ipc.c:424:P:ipc_exit_us:test_ipc_exit_us:0: Passed
> ../../tests/check_ipc.c:304:P:ipc_dispatch_us:test_ipc_disp_us:0: Passed
> ../../tests/check_ipc.c:654:P:ipc_bulk_events_us:test_ipc_bulk_events_us:0: Passed
> ../../tests/check_ipc.c:704:E:ipc_event_on_created_us:test_ipc_event_on_created_us:0: (after this point) Test timeout expired
> ../../tests/check_ipc.c:757:E:ipc_disconnect_after_created_us:test_ipc_disconnect_after_created_us:0: (after this point) Test timeout expired
> FAIL: ipc.test
> [..]
>
>
>>
>>
>>>> -Angus
>>>>
>>>>>
>>>>> Core was generated by `/opt/local/libexec/pacemaker/cib'.
>>>>> Program terminated with signal 6, Aborted.
>>>>> #0 0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> (gdb) bt
>>>>> #0 0xfffffd7fff0f061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> #1 0xfffffd7fff0e4ddd in thr_kill () from /lib/64/libc.so.1
>>>>> #2 0xfffffd7fff07a971 in raise () from /lib/64/libc.so.1
>>>>> #3 0xfffffd7fff0500a1 in abort () from /lib/64/libc.so.1
>>>>> #4 0xfffffd7fff0503f5 in _assert () from /lib/64/libc.so.1
>>>>> #5 0xfffffd7fc021274e in qb_ipcs_connection_unref () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #6 0x0000000000410438 in cib_shutdown ()
>>>>> #7 0xfffffd7fbfc5533f in crm_signal_dispatch (source=0x49be80,
>>>>> callback=<optimized out>, userdata=<optimized out>) at
>>>>> mainloop.c:203
>>>>> #8 0xfffffd7fc555f9e0 in g_main_context_dispatch () from
>>>>> /opt/local/lib/libglib-2.0.so.0
>>>>> #9 0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from
>>>>> /opt/local/lib/libglib-2.0.so.0 #10 0xfffffd7fc5560152 in
>>>>> g_main_loop_run () from /opt/local/lib/libglib-2.0.so.0
>>>>> #11 0x0000000000411056 in cib_init ()
>>>>> #12 0x000000000041163e in main ()
>>>>>
>>>>> Core was generated by `/opt/local/libexec/pacemaker/lrmd'.
>>>>> Program terminated with signal 6, Aborted.
>>>>> #0 0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> (gdb) bt
>>>>> #0 0xfffffd7fff0e061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> #1 0xfffffd7fff0d4ddd in thr_kill () from /lib/64/libc.so.1
>>>>> #2 0xfffffd7fff06a971 in raise () from /lib/64/libc.so.1
>>>>> #3 0xfffffd7fff0400a1 in abort () from /lib/64/libc.so.1
>>>>> #4 0xfffffd7fff0403f5 in _assert () from /lib/64/libc.so.1
>>>>> #5 0xfffffd7fc021274e in qb_ipcs_connection_unref () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #6 0xfffffd7fc02128a4 in qb_ipcs_disconnect () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #7 0xfffffd7fc0212995 in qb_ipcs_unref () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #8 0xfffffd7fc02129c7 in qb_ipcs_destroy () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #9 0xfffffd7fbfc55a3f in mainloop_del_ipc_server
>>>>> (server=<optimized out>) at mainloop.c:517 #10 0x00000000004041cd
>>>>> in lrmd_shutdown ()
>>>>> #11 0xfffffd7fbfc5533f in crm_signal_dispatch (source=0x48ad40,
>>>>> callback=<optimized out>, userdata=<optimized out>) at
>>>>> mainloop.c:203
>>>>> #12 0xfffffd7fc555f9e0 in g_main_context_dispatch () from
>>>>> /opt/local/lib/libglib-2.0.so.0
>>>>> #13 0xfffffd7fc555fd40 in g_main_context_iterate.isra.24 () from
>>>>> /opt/local/lib/libglib-2.0.so.0
>>>>> #14 0xfffffd7fc5560152 in g_main_loop_run () from
>>>>> /opt/local/lib/libglib-2.0.so.0
>>>>> #15 0x00000000004045e3 in main ()
>>>>>
>>>>> Core was generated by `/opt/local/libexec/pacemaker/stonithd'.
>>>>> Program terminated with signal 6, Aborted.
>>>>> #0 0xfffffd7fff11061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> (gdb) bt
>>>>> #0 0xfffffd7fff11061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> #1 0xfffffd7fff104ddd in thr_kill () from /lib/64/libc.so.1
>>>>> #2 0xfffffd7fff09a971 in raise () from /lib/64/libc.so.1
>>>>> #3 0xfffffd7fff0700a1 in abort () from /lib/64/libc.so.1
>>>>> #4 0xfffffd7fff0703f5 in _assert () from /lib/64/libc.so.1
>>>>> #5 0xfffffd7fc021274e in qb_ipcs_connection_unref () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #6 0xfffffd7fc02128a4 in qb_ipcs_disconnect () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #7 0xfffffd7fc0212995 in qb_ipcs_unref () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #8 0xfffffd7fc02129c7 in qb_ipcs_destroy () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #9 0x0000000000405e60 in ?? ()
>>>>> #10 0x0000000000407d28 in main ()
>>>>>
>>>>>
>>>>> PS:
>>>>> while reproducing the above, just caught the same with corosync (2.3.0):
>>>>>
>>>>> Core was generated by `corosync'.
>>>>> Program terminated with signal 6, Aborted.
>>>>> #0 0xfffffd7fff1e061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> (gdb) bt
>>>>> #0 0xfffffd7fff1e061a in _lwp_kill () from /lib/64/libc.so.1
>>>>> #1 0xfffffd7fff1d4ddd in thr_kill () from /lib/64/libc.so.1
>>>>> #2 0xfffffd7fff16a971 in raise () from /lib/64/libc.so.1
>>>>> #3 0xfffffd7fff1400a1 in abort () from /lib/64/libc.so.1
>>>>> #4 0xfffffd7fff1403f5 in _assert () from /lib/64/libc.so.1
>>>>> #5 0xfffffd7fc021274e in qb_ipcs_connection_unref () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #6 0xfffffd7fc02128a4 in qb_ipcs_disconnect () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #7 0xfffffd7fc0212995 in qb_ipcs_unref () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #8 0xfffffd7fc02129c7 in qb_ipcs_destroy () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #9 0x00000000004264e2 in cs_ipcs_service_destroy () #10
>>>>> 0x00000000004270e0 in ?? ()
>>>>> #11 0xfffffd7fc0210ff4 in job_dispatch () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #12 0xfffffd7fc020fcb4 in qb_loop_run () from
>>>>> /opt/local/lib/libqb.so.0
>>>>> #13 0x000000000042b551 in main ()
>>>>>
>>>>> so it looks like libqb related issue for me.
>>>>>
>>>>>
>>>>> Any help would be greatly appreciated!
>>>>>
>>>>>
>>>>> Best regards,
>>>>> Andrei
>>>>> _______________________________________________
>>>>> quarterback-devel mailing list
>>>>> quarterback-devel at lists.fedorahosted.org
>>>>> https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel
>>>> _______________________________________________
>>>> quarterback-devel mailing list
>>>> quarterback-devel at lists.fedorahosted.org
>>>> https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel
>>>
>>
>
> _______________________________________________
> quarterback-devel mailing list
> quarterback-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel
> _______________________________________________
> quarterback-devel mailing list
> quarterback-devel at lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel
More information about the quarterback-devel
mailing list