[libqb] libqb and multy-thread application

Angus Salkeld asalkeld at redhat.com
Tue Apr 2 07:26:52 UTC 2013


On 02/04/13 10:57 +0400, eXeC001er wrote:
>2013/4/2 Angus Salkeld <asalkeld at redhat.com>
>
>> On 29/03/13 01:13 +0400, eXeC001er wrote:
>>
>>> Hello.
>>>
>>> This Q is crosspost from openais-maillist.
>>>
>>> I use corosync-2.3.0, libqb-0.14.4 on OpenIndiana:
>>>
>>> I tried to create an application that uses corosync via its libraries, but
>>> it seems something wrong.
>>> On the machine where i debug corosync-tests work fine, but my application
>>> does not work. The application cannot intialize CMAP connection (same with
>>> VOTEQUORUM).
>>> "*_initialize" function returned CS_ERR_NOT_EXIST error. on server side i
>>> see "Error in connection setup (7104-7367-0): Broken pipe (32)".
>>>
>>> My appliacation is multi-thread and "*_initialize" is called not in main
>>> thread.
>>>
>>
>> Well, I suggest you keep it simple initially.
>> Do the standard corosync apps work ok?
>> like:
>> ./tests/testcpg
>>
>
>yes. all tests from test directory of corosync sources work fine.

Well then there is no reason why other apps should not work.
Just have a look at how they are written.

>
>
>>
>> It would be good to try narrow down what is failing.
>>
>>
>>
>>> the stack-trace:
>>>
>>> /3 at 3:   -> libvotequorum:votequorum_**initialize(0x8834b2c, 0x0, 0x0,
>>> 0xfea63855)
>>> /2: nanosleep(0xFE99BF48, 0x00000000)       = 0
>>> /3 at 3:     -> libqb:qb_hdb_handle_create()
>>> /3 at 3:     <- libqb:qb_hdb_handle_create() = 0
>>> /3 at 3:     -> libcorosync_common:hdb_error_**to_cs()
>>> /3 at 3:     <- libcorosync_common:hdb_error_**to_cs() = 1
>>> /3 at 3:     -> libqb:qb_hdb_handle_get()
>>> /3 at 3:     <- libqb:qb_hdb_handle_get() = 0
>>> /3 at 3:     -> libcorosync_common:hdb_error_**to_cs()
>>> /3 at 3:     <- libcorosync_common:hdb_error_**to_cs() = 1
>>> /3 at 3:     -> libqb:qb_ipcc_connect()
>>> /3: so_socket(PF_UNIX, SOCK_STREAM, 0, 0x00000000, SOV_DEFAULT) = 8
>>> /3: fcntl(8, F_GETFD, 0x00000000)           = 0
>>> /3: fcntl(8, F_SETFD, 0x00000001)           = 0
>>> /3: fcntl(8, F_SETFL, FNONBLOCK)            = 0
>>> /3: connect(8, 0xFE67C13A, 21, SOV_DEFAULT)     = 0
>>> /3: sigaction(SIGPIPE, 0xFE67C0E0, 0xFE67C140)  = 0
>>> /3: send(8, "FFFFFFFF\0\0\0\018\0\0\0".., 24, 0)    = 24
>>> /3: sigaction(SIGPIPE, 0xFE67C0E0, 0x00000000)  = 0
>>> /3: sigaction(SIGPIPE, 0xFE67C0D0, 0xFE67C130)  = 0
>>> /3: recv(8, 0xFE67C210, 3112, 64)           Err#11 EAGAIN
>>>
>>
>> this ^ is the client returngin EAGAIN on the socket...
>> the problem is probably on the server.
>
>
>on server side i also see strange case:
>
>if i try to run a corosync-* tool, then the tool returned required info,
>but on server side i see "File not found" error. I see that the corosync
>daemon tries to do something with /proc/<client_pid> file:
>
>an example:
>
>......... [QB    ] recv from client connection failed (10656-12155-22): No
>such file or directory (2)
>

I don't think this is a problem, oindiana/solaris sockets behave a bit
differently to linux ones and the path to finding a disconnection are
just a bit different. I think this is just using the wrong errno.

The "/proc/<pid>/" access is also fine, it tries on each connection
to resolve the process name, and if it can't just uses the pid.


>
>>
>>
>>  /3: sigaction(SIGPIPE, 0xFE67C0D0, 0x00000000)  = 0
>>> /3: shutdown(8, SHUT_RDWR, SOV_DEFAULT)     = 0
>>> /3: close(8)                    = 0
>>> /3 at 3:     <- libqb:qb_ipcc_connect() = 0
>>> /3 at 3:     -> libcorosync_common:qb_to_cs_**error()
>>> /3 at 3:     <- libcorosync_common:qb_to_cs_**error() = 12
>>> /3 at 3:     -> libqb:qb_hdb_handle_put(**0xfe9f3070, 0x0, 0xb86775,
>>> 0xfe67ce7c)
>>> /3 at 3:     <- libqb:qb_hdb_handle_put() = 0
>>> /3 at 3:     -> libqb:qb_hdb_handle_destroy(**0xfe9f3070, 0x0, 0xb86775,
>>> 0x0)
>>> /3 at 3:       -> libqb:qb_ipcc_disconnect(0x0, 0x8835b88, 0x0, 0x1)
>>> /3 at 3:       <- libqb:qb_ipcc_disconnect() = 0
>>> /3 at 3:     <- libqb:qb_hdb_handle_destroy() = 0
>>> /3 at 3:   <- libvotequorum:votequorum_**initialize() = 12
>>>
>>> As you can see it returns CS_ERR_NOT_EXIST.
>>>
>>> same behaviour with CMAP and another services.
>>>
>>> Any ideas?
>>>
>>
>> Can you turn on debug in corosync.conf and to_stderr
>> logging {
>> to_stderr: yes
>> debug: on
>> }
>> and run corosync like this:
>> "sudo corosync -f"
>>
>> then in another window run:
>> ./tests/testcpg
>>
>> then type a couple of words (any junk) - testcpg should send what ever
>> is typed to corosync. Then Ctrl-C to end.
>>
>> Can you send the contents of both shells so I can see what both the
>> client and server are doing?
>>
>
>CLIENT:
>
>$ sudo ./testcpg
>Type EXIT to finish
>Local node id is ac12fe0a
>membership list
>nza4-devel:20446 2013-04-02 10:43:38.902
>ConfchgCallback: group 'GROUP'
>Local node id is 10.254.18.172/ac12fe0a result 1
>joined node/pid -1408041462/20446 reason: 1
>nodes in group now 1
>node/pid -1408041462/20446
>
>nza4-devel:20446 2013-04-02 10:43:38.904
>TotemConfchgCallback: ringid (2886925834.156)
>active processors 1: -1408041462
>param param param param
>nza4-devel:20446 2013-04-02 10:44:07.682
>DeliverCallback: message (len=25)from node/pid -1408041462/20446: 'param
>param param param
>'
>
>
>SERVER:
>
>see attached file or http://pastie.org/7276133
>

I do have access to a oindiana machine if you have a test app that
fails can you send it?

-Angus

>
>>
>> -Angus
>>
>>  Thanks.
>>>
>>
>>  ______________________________**_________________
>>> quarterback-devel mailing list
>>> quarterback-devel at lists.**fedorahosted.org<quarterback-devel at lists.fedorahosted.org>
>>> https://lists.fedorahosted.**org/mailman/listinfo/**quarterback-devel<https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel>
>>>
>>
>> ______________________________**_________________
>> quarterback-devel mailing list
>> quarterback-devel at lists.**fedorahosted.org<quarterback-devel at lists.fedorahosted.org>
>> https://lists.fedorahosted.**org/mailman/listinfo/**quarterback-devel<https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel>
>>


>_______________________________________________
>quarterback-devel mailing list
>quarterback-devel at lists.fedorahosted.org
>https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel



More information about the quarterback-devel mailing list