[libqb] libqb and multy-thread application

eXeC001er execooler at gmail.com
Wed Apr 3 12:16:13 UTC 2013


have found the following:

1. I compiled corosync and libqb without '-pthread'

2. thread stack size on Solaris:
    platform 32 bit  = 1 MB
    platform 64 bit = 2 MB

The second case is related to corosync, because it
defines IPC_DISPATCH_SIZE = 1 MB without '--enable-small-memory-footprint'.

Thanks.

2013/4/2 eXeC001er <execooler at gmail.com>

>
>
> 2013/4/2 Angus Salkeld <asalkeld at redhat.com>
>
>> On 02/04/13 10:57 +0400, eXeC001er wrote:
>>
>>> 2013/4/2 Angus Salkeld <asalkeld at redhat.com>
>>>
>>>  On 29/03/13 01:13 +0400, eXeC001er wrote:
>>>>
>>>>  Hello.
>>>>>
>>>>> This Q is crosspost from openais-maillist.
>>>>>
>>>>> I use corosync-2.3.0, libqb-0.14.4 on OpenIndiana:
>>>>>
>>>>> I tried to create an application that uses corosync via its libraries,
>>>>> but
>>>>> it seems something wrong.
>>>>> On the machine where i debug corosync-tests work fine, but my
>>>>> application
>>>>> does not work. The application cannot intialize CMAP connection (same
>>>>> with
>>>>> VOTEQUORUM).
>>>>> "*_initialize" function returned CS_ERR_NOT_EXIST error. on server
>>>>> side i
>>>>> see "Error in connection setup (7104-7367-0): Broken pipe (32)".
>>>>>
>>>>> My appliacation is multi-thread and "*_initialize" is called not in
>>>>> main
>>>>> thread.
>>>>>
>>>>>
>>>> Well, I suggest you keep it simple initially.
>>>> Do the standard corosync apps work ok?
>>>> like:
>>>> ./tests/testcpg
>>>>
>>>>
>>> yes. all tests from test directory of corosync sources work fine.
>>>
>>
>> Well then there is no reason why other apps should not work.
>> Just have a look at how they are written.
>>
>
> Do not forget that tests are single-thread app.
> My app is multi-thread and initialization not in main-thread. if i do
> initialization in main-thread then everything fine.
>
>
>
>
>>
>>
>>>
>>>
>>>> It would be good to try narrow down what is failing.
>>>>
>>>>
>>>>
>>>>  the stack-trace:
>>>>>
>>>>> /3 at 3:   -> libvotequorum:votequorum_****initialize(0x8834b2c, 0x0,
>>>>> 0x0,
>>>>>
>>>>> 0xfea63855)
>>>>> /2: nanosleep(0xFE99BF48, 0x00000000)       = 0
>>>>> /3 at 3:     -> libqb:qb_hdb_handle_create()
>>>>> /3 at 3:     <- libqb:qb_hdb_handle_create() = 0
>>>>> /3 at 3:     -> libcorosync_common:hdb_error_****to_cs()
>>>>> /3 at 3:     <- libcorosync_common:hdb_error_****to_cs() = 1
>>>>>
>>>>> /3 at 3:     -> libqb:qb_hdb_handle_get()
>>>>> /3 at 3:     <- libqb:qb_hdb_handle_get() = 0
>>>>> /3 at 3:     -> libcorosync_common:hdb_error_****to_cs()
>>>>> /3 at 3:     <- libcorosync_common:hdb_error_****to_cs() = 1
>>>>>
>>>>> /3 at 3:     -> libqb:qb_ipcc_connect()
>>>>> /3: so_socket(PF_UNIX, SOCK_STREAM, 0, 0x00000000, SOV_DEFAULT) = 8
>>>>> /3: fcntl(8, F_GETFD, 0x00000000)           = 0
>>>>> /3: fcntl(8, F_SETFD, 0x00000001)           = 0
>>>>> /3: fcntl(8, F_SETFL, FNONBLOCK)            = 0
>>>>> /3: connect(8, 0xFE67C13A, 21, SOV_DEFAULT)     = 0
>>>>> /3: sigaction(SIGPIPE, 0xFE67C0E0, 0xFE67C140)  = 0
>>>>> /3: send(8, "FFFFFFFF\0\0\0\018\0\0\0".., 24, 0)    = 24
>>>>> /3: sigaction(SIGPIPE, 0xFE67C0E0, 0x00000000)  = 0
>>>>> /3: sigaction(SIGPIPE, 0xFE67C0D0, 0xFE67C130)  = 0
>>>>> /3: recv(8, 0xFE67C210, 3112, 64)           Err#11 EAGAIN
>>>>>
>>>>>
>>>> this ^ is the client returngin EAGAIN on the socket...
>>>> the problem is probably on the server.
>>>>
>>>
>>>
>>> on server side i also see strange case:
>>>
>>> if i try to run a corosync-* tool, then the tool returned required info,
>>> but on server side i see "File not found" error. I see that the corosync
>>> daemon tries to do something with /proc/<client_pid> file:
>>>
>>> an example:
>>>
>>> ......... [QB    ] recv from client connection failed (10656-12155-22):
>>> No
>>> such file or directory (2)
>>>
>>>
>> I don't think this is a problem, oindiana/solaris sockets behave a bit
>> differently to linux ones and the path to finding a disconnection are
>> just a bit different. I think this is just using the wrong errno.
>>
>> The "/proc/<pid>/" access is also fine, it tries on each connection
>> to resolve the process name, and if it can't just uses the pid.
>>
>>
>>
>>>
>>>>
>>>>  /3: sigaction(SIGPIPE, 0xFE67C0D0, 0x00000000)  = 0
>>>>
>>>>> /3: shutdown(8, SHUT_RDWR, SOV_DEFAULT)     = 0
>>>>> /3: close(8)                    = 0
>>>>> /3 at 3:     <- libqb:qb_ipcc_connect() = 0
>>>>> /3 at 3:     -> libcorosync_common:qb_to_cs_****error()
>>>>> /3 at 3:     <- libcorosync_common:qb_to_cs_****error() = 12
>>>>> /3 at 3:     -> libqb:qb_hdb_handle_put(****0xfe9f3070, 0x0, 0xb86775,
>>>>>
>>>>> 0xfe67ce7c)
>>>>> /3 at 3:     <- libqb:qb_hdb_handle_put() = 0
>>>>> /3 at 3:     -> libqb:qb_hdb_handle_destroy(****0xfe9f3070, 0x0,
>>>>> 0xb86775,
>>>>>
>>>>> 0x0)
>>>>> /3 at 3:       -> libqb:qb_ipcc_disconnect(0x0, 0x8835b88, 0x0, 0x1)
>>>>> /3 at 3:       <- libqb:qb_ipcc_disconnect() = 0
>>>>> /3 at 3:     <- libqb:qb_hdb_handle_destroy() = 0
>>>>> /3 at 3:   <- libvotequorum:votequorum_****initialize() = 12
>>>>>
>>>>>
>>>>> As you can see it returns CS_ERR_NOT_EXIST.
>>>>>
>>>>> same behaviour with CMAP and another services.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>>
>>>> Can you turn on debug in corosync.conf and to_stderr
>>>> logging {
>>>> to_stderr: yes
>>>> debug: on
>>>> }
>>>> and run corosync like this:
>>>> "sudo corosync -f"
>>>>
>>>> then in another window run:
>>>> ./tests/testcpg
>>>>
>>>> then type a couple of words (any junk) - testcpg should send what ever
>>>> is typed to corosync. Then Ctrl-C to end.
>>>>
>>>> Can you send the contents of both shells so I can see what both the
>>>> client and server are doing?
>>>>
>>>>
>>> CLIENT:
>>>
>>> $ sudo ./testcpg
>>> Type EXIT to finish
>>> Local node id is ac12fe0a
>>> membership list
>>> nza4-devel:20446 2013-04-02 10:43:38.902
>>> ConfchgCallback: group 'GROUP'
>>> Local node id is 10.254.18.172/ac12fe0a result 1
>>> joined node/pid -1408041462/20446 reason: 1
>>> nodes in group now 1
>>> node/pid -1408041462/20446
>>>
>>> nza4-devel:20446 2013-04-02 10:43:38.904
>>> TotemConfchgCallback: ringid (2886925834.156)
>>> active processors 1: -1408041462
>>> param param param param
>>> nza4-devel:20446 2013-04-02 10:44:07.682
>>> DeliverCallback: message (len=25)from node/pid -1408041462/20446: 'param
>>> param param param
>>> '
>>>
>>>
>>> SERVER:
>>>
>>> see attached file or http://pastie.org/7276133
>>>
>>>
>> I do have access to a oindiana machine if you have a test app that
>> fails can you send it?
>
>
> will try later.
>
>
>>
>> -Angus
>>
>>
>>>
>>>> -Angus
>>>>
>>>>  Thanks.
>>>>
>>>>>
>>>>>
>>>>  ______________________________****_________________
>>>>
>>>>> quarterback-devel mailing list
>>>>> quarterback-devel at lists.**fedo**rahosted.org <http://fedorahosted.org>
>>>>> <quarterback-**devel at lists.fedorahosted.org<quarterback-devel at lists.fedorahosted.org>
>>>>> >
>>>>> https://lists.fedorahosted.****org/mailman/listinfo/****
>>>>> quarterback-devel<https://**lists.fedorahosted.org/**
>>>>> mailman/listinfo/quarterback-**devel<https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel>
>>>>> >
>>>>>
>>>>>
>>>> ______________________________****_________________
>>>> quarterback-devel mailing list
>>>> quarterback-devel at lists.**fedo**rahosted.org <http://fedorahosted.org><
>>>> quarterback-**devel at lists.fedorahosted.org<quarterback-devel at lists.fedorahosted.org>
>>>> >
>>>> https://lists.fedorahosted.****org/mailman/listinfo/****
>>>> quarterback-devel<https://**lists.fedorahosted.org/**
>>>> mailman/listinfo/quarterback-**devel<https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel>
>>>> >
>>>>
>>>>
>>
>>  ______________________________**_________________
>>> quarterback-devel mailing list
>>> quarterback-devel at lists.**fedorahosted.org<quarterback-devel at lists.fedorahosted.org>
>>> https://lists.fedorahosted.**org/mailman/listinfo/**quarterback-devel<https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel>
>>>
>>
>> ______________________________**_________________
>> quarterback-devel mailing list
>> quarterback-devel at lists.**fedorahosted.org<quarterback-devel at lists.fedorahosted.org>
>> https://lists.fedorahosted.**org/mailman/listinfo/**quarterback-devel<https://lists.fedorahosted.org/mailman/listinfo/quarterback-devel>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.fedorahosted.org/pipermail/quarterback-devel/attachments/20130403/91f54b07/attachment.html>


More information about the quarterback-devel mailing list