[Libwebsockets] Interaction of external threads with libwebsockets server

"Andy Green (林安廸)" andy at warmcat.com
Thu Dec 19 12:29:16 CET 2013


On 19/12/13 02:12, the mail apparently from Thomas Spitz included:
> Hello Andy, hello Eugene,
>
> If I understood it correctly, thanks to your patch, we can now prevent
> our external thread(s) from calling libwebsocket_callback_on_writable
> while lws is actively dealing with POLLIN and POLLOUT of fds using our
> own mutex? Thus we are sure that the request(s) from our external
> thread(s) will not be lost, or at least trigger
> a LWS_CALLBACK_SERVER_WRITEABLE?

I'm not sure it won't get lost, but it won't leave things in an 
indeterminate state.  It can still happen the non-service thread asks to 
set POLLOUT just as we're about to take the mutex and clear it.

At the moment that'll still break at low probability but it should be 
solveable.

> My second question is related to performance:
>
> In the documentation it is said
>
>     If you want to send something, do not just send it but request a
>     callback |when the socket is writeable |
>
> That is why our external thread call libwebsocket_callback_on_writable
> (). Then it is written
>
>     |Usually you will get called back immediately next time around the
>     service loop|
>
> which seems to mean that  (according to my tests) we have to
> wait libwebsocket_service() to end up before the socket is writeable
> again. That implies to set a very short timeout

Well... you are right if we have one connection.  But if you have 50 or 
50,000 connections, somebody almost always has something going on before 
the timeout.

Also in your user callback, you can take the approach like in the test 
apps to loop using lws_send_pipe_choked() test to absolutely stuff your 
pipe each time you get service.  That doesn't delay anything looping 
there because it only loops while the send won't block.  And it means 
the kernel is passing out what you stuffed into the pipe while you wait 
through any latency in the next "writeable" service.

> for libwebsocket_service(context, VERYSHORTTIMEOUT) otherwise it will
> take a long time before the external thread request could be handled...
> This brings me to wonder why it is not possible to loop on
> libwebsocket_service(context, VERYLONGTIMEOUT) that would be interrupted
> by our external thread instead? Thus lws thread is asleeped most of the
> time when there is no incoming IP traffic but can be awaken as soon as
> some outgoing traffic needs to be sent (generated from the external thread).

Well lws is a singlethreaded nonblocking implementation.  It is designed 
to run on small resource-costrained platforms at the same time a being 
able to service huge numbers of connections in a lightweight way.  It 
does not require threads.

Poll can only wait on file or socket descriptors... you could reserve a 
socket to signal to break out and reload the poll() because somebody's 
attribute changed.  But in most real usage scenarios, some other 
connection is always wanting service before the timeout anyway and as 
more connections are seen the less it matters about latency from waiting 
for poll() reload.

-Andy

> I hope my two questions are clear..
>
> Thanks in advance for your rely.
>
> Best regards, Thomas
>
> 2013/12/18 "Andy Green (林安廸)" <andy at warmcat.com
> <mailto:andy at warmcat.com>>
>
>     On 18/12/13 06:40, the mail apparently from Andy Green included:
>
>
>
>         Eugene Agafonov <e.a.agafonov at gmail.com
>         <mailto:e.a.agafonov at gmail.com>> wrote:
>
>             Hi!
>
>             I've just implemented the similar scenario just like it is
>             stated
>             in document
>             http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__tree/README.coding
>             <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/tree/README.coding>
>
>
>
>     I have a mutex-protected queue of outgoing events (as a std::list of
>
>             strings). Each string is intended to be send in separated
>             message.
>
>             The writer thread posts data onto queue and calls
>             libwebsocket_callback_on___writable(context, wsi)
>
>             Protocol callback gets LWS_CALLBACK_SERVER_WRITEABLE as soon as
>             websocket thread is ready to send data. Once it comes, the
>             callback
>             extracts strings from the queue one by one and writes them to
>             socket using libwebsocket_write.
>
>             I'm quite newby in writing WS server with libwebsockets. Any
>             comments from WS gurus are welcome
>
>
>         What you're doing is a good way to interact with what lws needs and
>         get what you want from threading.
>
>         There's one danger with it, but that is a very tightly defined race
>         rather than corruption or whatever.
>
>
>     Well... there's another thing to take care about... you need to
>     manage your private list of live wsis carefully.  It means managing
>     that list (with any of your locking needed) at the ESTABLISHED /
>     CLOSED callback for the WSI.  That should be enough I think.
>
>
>         The issue is that the 'wait for writeable' is oneshot, so after it
>         becomes true in the poll lws will clear the POLLOUT wait flag.  But
>         because the threads are asynchronous, eventually the other thread
>         will come at the wrong time and set it just as we're clearing it
>         with
>         indeterminate results.  Because it's read-modify-write and flow
>         control code also modifies this, there's a low probability deadlock
>         case where POLLIN or POLLOUT from one thread or another did not get
>         set for wait as the thread thinks it is.
>
>         If the extpoll stuff is used the issue can be worse depending on
>         what
>         it is doing.
>
>         However the approach you're taking is otherwise really workable I
>         think.  To make it rock solid you'll need to patch lws to
>         mutex-protect the few places it changes poll wait flags.
>
>
>     I added a patch
>
>     http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e
>     <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=7a1327977ac10bdbace0012274b8ae889219880e>
>
>     that lets you do this in a lock and unlock callback in the user
>     code, in one place.
>
>     If you're not using threading you don't need to do anything new.
>
>     -Andy
>
>
>         -Andy
>
>             BR, Eugene Agafonov.
>
>             On Tuesday 17 December 2013 12:03:43 Thomas Spitz wrote:
>
>                 Hello everyone,
>
>                 I am trying to find a way for external threads to ask
>                 libwebsocket
>
>             server
>
>                 to send websocket message on their behalf (as they
>                 cannot do it
>
>             directly).
>
>                 The only way I see at the present is to make a loop with
>                 libwebsocket_service (context, VERYSHORTTIME); and test
>                 potential
>
>             incoming
>
>                 messages from other threads. This is not very elegant as CPU
>                 computes
>
>             all
>
>                 the time and not very reliable as I should put
>                 VERYSHORTTIME at 0
>                 if
>
>             I
>
>                 don't want to miss any message.
>
>                 I tried the following modification in test-server.c :
>
>                 void *connexionServeur(void *arg) {
>
>                     struct libwebsocket_context *context = arg; while (1) {
>                     if(libwebsocket_service(__context, 60000)<0){
>
>                     printf("Problem!\n");
>
>                     break; } }
>
>                     return NULL ;
>
>                     } int statut_Thread; pthread_t Thread; statut_Thread =
>                     pthread_create(&Thread, NULL, connexionServeur,
>
>             context
>
>                     );
>
>                     if (statut_Thread != 0) {
>
>                     printf("Problem with thread creation \n"); } while
>                     (n >= 0 &&
>                     !force_exit) { struct timeval tv; gettimeofday(&tv,
>                     NULL); if
>                     (((unsigned int)tv.tv_usec - oldus) > 50000) {
>
>
>             libwebsocket_callback_on___writable_all_protocol(&__protocols[PROTOCOL_DUMB_I
>
>
>
>     NCREMENT]); oldus = tv.tv_usec;
>
>                     libwebsocket_service(context, 0);
>
>                     }
>
>                     usleep(10000); }
>
>
>                 but it crashes quickly as soon as I try drawing . I
>                 suppose that
>                 libwebsocket_service is called twice at the same time and it
>                 creates conflicts that lead to stop crash the websocket.
>
>                 I have seen in the mailing list that Thomas Koeper has
>                 already
>                 had a similar question
>
>             http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-__April/000403.html
>             <http://ml.libwebsockets.org/pipermail/libwebsockets/2013-April/000403.html>
>
>
>
>     . At that time, he was thinking of creating a mutex on
>
>                 libwebsocket_service. Do you think this is the right
>                 solution?
>
>                 Thanks in advance for your help Best regards, Thomas
>
>             _________________________________________________ Libwebsockets
>             mailing list Libwebsockets at ml.__libwebsockets.org
>             <mailto:Libwebsockets at ml.libwebsockets.org>
>             http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>             <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>
>
>
>         _________________________________________________ Libwebsockets
>         mailing
>         list Libwebsockets at ml.__libwebsockets.org
>         <mailto:Libwebsockets at ml.libwebsockets.org>
>         http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>
>
>
>     _________________________________________________
>     Libwebsockets mailing list
>     Libwebsockets at ml.__libwebsockets.org
>     <mailto:Libwebsockets at ml.libwebsockets.org>
>     http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>     <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>
>
>




More information about the Libwebsockets mailing list