[Libwebsockets] Interaction of external threads with libwebsockets server

"Andy Green (林安廸)" andy at warmcat.com
Fri Jan 3 06:18:59 CET 2014


On 01/01/14 22:38, the mail apparently from Andy Green included:
>
>
> Thomas Spitz <thomas.spitz at hestia-france.com> wrote:
>> Hello Andy,
>>
>> Have you had some time trying to replace poll by ppoll in order to have
>> poll triggered on signal from an external thread?
>
> Not yet... in Taiwan the big holiday is Chinese New Year in a few weeks.  I'm still interested in doing it, the weekend is the most likely time.

Please have a look at this:

http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=3b3fa9e2086da6157289141e0b6fe1e5035bad25

I didn't test it because I don't have a threaded user code, but it 
should be pretty close if not workable already.

Note the comment in the commit log, you have to actively enable this 
code (I wasn't able to find a way for the compiler to understand if it 
had ppoll() or not).

ppoll() is a GNU extension so if this is useful, we'll need to add it as 
a CMake-time option.

-Andy


> -Andy
>
>> At the present your lib works very well with intensive data submit from
>> external thread.
>>
>> Happy new year to everyone.
>>
>> BR,
>>
>> Thomas
>>
>>
>> On 25 Dec 2013 15:24, Andy Green (林安廸) <andy at warmcat.com> wrote:
>>
>>> On 25/12/13 20:14, the mail apparently from Thomas Spitz included:
>>>
>>>>      The choices seem to boil down to this kind of "add a fake
>>>>      descriptor" thing (although everything, including the "interrupt
>> the
>>>>      poll" descriptor and the use of it should be defined inside the
>>>>      library), or maybe change to use ppoll() and fire signals at it.
>>>>
>>>> ppoll() could be an interesting solution but it only interrupt the
>> poll.
>>>>
>>>
>>> I think that's all we need to do.
>>>
>>> Latency is only coming this way on an idle system where we are
>> sleeping in
>>> the poll(), but another thread asked to change what a pollfd was
>> waiting on.
>>>
>>> As you pointed out originally, under those circumstances the changed
>>> pollfd rules won't be seen and handled -- if every fd is idle for the
>>> events it started out with -- until the poll() timeout expires.
>>>
>>> Although in other use-cases this isn't that realistic as a problem,
>> since
>>> some deal with tens of thousands of simultaneous connections and
>> usually
>>> someone is breaking the poll after a short time for service, in other
>> use
>>> cases it is realistic.  You can attack it by reducing the poll sleep
>> period
>>> but then you're looking at maybe hundreds of wakes a second on what
>> should
>>> be an idle system, needlessly bad for power.
>>>
>>> If we provided a way for those use-cases to have very long poll()
>> timeouts
>>> and minimal latency it's good I think, so long as it doesn't burden
>> or make
>>> problems when it's not wanted or needed.
>>>
>>>   I was thinking of interrupting poll() using a named pipe in which I
>>>> would have told lws which wsi it needs to write to. The complete
>> process
>>>> would have been the following:
>>>>
>>>
>>> No it's not a good way... lws already has a good semantic in poll()
>> for
>>> understanding who needed service.  This would be a lot of new stuff
>> doing
>>> the same job that only works in the multithreaded case.
>>>
>>>   1) Before libwebsocket_create_context(), I create the named pipe.
>>>> 2) For every client connection, I book for a shm
>>>> in LWS_CALLBACK_ESTABLISHED through which I will share incoming data
>>>> with my main thread
>>>> 3) My main thread process the incoming data and store the answer
>> into
>>>> the shm. It then indicates lws that an answer is ready for a given
>> wsi
>>>> indicating the ID of the shm in the named pipe
>>>> 4) lws poll() is interupted and it knows immediatly which wsi it
>> needs
>>>> to write to thanks to the ID of the shm. If the wsi is closed in the
>>>> meanwhile, lws indicate it to the shm in LWS_CALLBACK_CLOSED
>>>>
>>>> If I use ppoll(), I could keep almost the same principle but I would
>>>> then need to add a SIGUSR1 and a handler OR loop through my client
>> shm
>>>> array each time ppoll() got interupted with EINTR flag set...
>> Finally I
>>>> am still wondering whether my solution is not simpler?
>>>>
>>>
>>> That solution is basically a threaded rewrite of lws not using
>> poll(). If
>>> you're interested to do that I don't want to discourage you, but it's
>>> something different from lws then.  Of course lws is liberally
>> licensed so
>>> you're welcome to build on it if you have a compatible license.
>>>
>>> However, if you think about larger scale servers, which do exist
>> using
>>> lws, "knowing the exact (single) wsi" that woke it is not useful when
>> there
>>> may be hundreds of fds needing service each poll().
>>>
>>>       Either way lws_change_pollfd() is central to the solution.
>>>>
>>>> With my solution or even ppoll one, I don't see when I need to
>>>> calllws_change_pollfd() especially as lws_change_pollfd needs a
>> pointer
>>>> to wsi which I cannot give as my interrupt concerns the complete
>> context
>>>> and not a special wsi...?I must miss a point.
>>>>
>>>
>>> lws_change_pollfd() is the point that any code which wants to change
>> the
>>> events on a pollfd ends up at now.  And changing the event on a
>> pollfd is
>>> the definition of the cause of latency (when poll() is idle and with
>>> relatively long timeout).
>>>
>>> So whether it is doing rx flow control or wait on being able to send,
>> that
>>> function is the place to signal to break the poll() one way or the
>> other.
>>>
>>>       If you pick a signal like SIGUSR1 and install a do-nothing
>> handler
>>>>      for it, firing SIGUSR1 at the process from itself in
>>>>      lws_change_pollfd() and using ppoll() could be a really small
>> and
>>>>      robust solution.
>>>>      Since the signal is handled it doesn't do anything except
>> interrupt
>>>>      the ppoll causing a pollfd reload.
>>>>      You only need to fire the signal the first time anything wants
>> to
>>>>      interrupt the wait *from another thread* (because if the lws
>> thread
>>>>      is in poll(), it isn't doing anything else).  If a pollfd raced
>> it
>>>>      and changed first, there's no problem with an additional signal
>>>>      interrupting the next ppoll loop.
>>>>
>>>> Ideally, if it is not too much to ask, a simple example of code
>> would be
>>>> ideal.
>>>>
>>>
>>> I may have some time tomorrow to give this a try.
>>>
>>> -Andy




More information about the Libwebsockets mailing list