[Libwebsockets] Interaction of external threads with libwebsockets server

Thomas Spitz thomas.spitz at hestia-france.com
Fri Dec 20 12:38:49 CET 2013


Hello Andy,

In fact, in my application, I mostly often only have 0 to 5 active
connections at the same time with a maximum number of simultaneous
connections restricted to 32 (Besides, why is there no simple solution to
reduce the maximum simultaneous connection? I have seen that we need to
reduce the max system fds...).

Therefore, I think I should implement your following proposal using
External_Poll I guess:

> Poll can only wait on file or socket descriptors... you could reserve a
> socket to signal to break out and reload the poll() because somebody's
> attribute changed.


I tried to do it but my thread is still not able to break out the poll().
In my test below (using test-server.c as skeletton),  Although I write in
the tube that is monitored by poll I see that my asyncSending is never able
to break up poll(). poll() always break up on 10s timeout whereas my thread
write into the tube every second.

int tube[2];

(...)

void *asyncSending(void *arg) {

unsigned char c=0;
 printf("START asynchronous sending\n");

while(1){
c++;
 printf("write in tube\n");
if(write(tube[1],&c,1)!=1){
printf("Erreur write %s\n", strerror(errno));
 }
//
libwebsocket_callback_on_writable_all_protocol(&protocols[PROTOCOL_DUMB_INCREMENT]);
}

printf("END asynchronous sending\n");
return NULL ;

}


 int main(int argc, char **argv) {

        (...)

n = 0;

 printf("Tube creation\n");
if(pipe(tube)!=0){
printf("Pipe creation error %s\n", strerror(errno));
 }

if (count_pollfds >= max_poll_elements) {
lwsl_err("LWS_CALLBACK_ADD_POLL_FD: too many sockets to track\n");
 return 1;
}

fd_lookup[tube[0]] = count_pollfds;
 pollfds[count_pollfds].fd = fd_lookup[tube[0]];
pollfds[count_pollfds].events = POLLIN | POLLPRI |POLL_OUT;
 pollfds[count_pollfds++].revents = 0;

int statut_Thread;
pthread_t Thread;

statut_Thread = pthread_create(&Thread, NULL, asyncSending, context);
if (statut_Thread != 0) {
 printf("Thread creation errror \n");
}

while (n >= 0 && !force_exit) {

/*
 * this represents an existing server's single poll action
 * which also includes libwebsocket sockets
 */

n = poll(pollfds, count_pollfds, 10000);//10s
if (n < 0)
 continue;

if (n)
for (n = 0; n < count_pollfds; n++)
 if (pollfds[n].revents)
/*
 * returns immediately if the fd does not
 * match anything under libwebsockets
 * control
 */
 if (libwebsocket_service_fd(context,
&pollfds[n]) < 0)
goto done;

               (...)
       }
}

Thanks again in advance for your help.

BR, Thomas


2013/12/19 "Andy Green (林安廸)" <andy at warmcat.com>

> On 19/12/13 02:12, the mail apparently from Thomas Spitz included:
>
>  Hello Andy, hello Eugene,
>>
>> If I understood it correctly, thanks to your patch, we can now prevent
>> our external thread(s) from calling libwebsocket_callback_on_writable
>> while lws is actively dealing with POLLIN and POLLOUT of fds using our
>> own mutex? Thus we are sure that the request(s) from our external
>> thread(s) will not be lost, or at least trigger
>> a LWS_CALLBACK_SERVER_WRITEABLE?
>>
>
> I'm not sure it won't get lost, but it won't leave things in an
> indeterminate state.  It can still happen the non-service thread asks to
> set POLLOUT just as we're about to take the mutex and clear it.
>
> At the moment that'll still break at low probability but it should be
> solveable.
>
>
>  My second question is related to performance:
>>
>> In the documentation it is said
>>
>>     If you want to send something, do not just send it but request a
>>     callback |when the socket is writeable |
>>
>> That is why our external thread call libwebsocket_callback_on_writable
>> (). Then it is written
>>
>>     |Usually you will get called back immediately next time around the
>>     service loop|
>>
>> which seems to mean that  (according to my tests) we have to
>> wait libwebsocket_service() to end up before the socket is writeable
>> again. That implies to set a very short timeout
>>
>
> Well... you are right if we have one connection.  But if you have 50 or
> 50,000 connections, somebody almost always has something going on before
> the timeout.
>
> Also in your user callback, you can take the approach like in the test
> apps to loop using lws_send_pipe_choked() test to absolutely stuff your
> pipe each time you get service.  That doesn't delay anything looping there
> because it only loops while the send won't block.  And it means the kernel
> is passing out what you stuffed into the pipe while you wait through any
> latency in the next "writeable" service.
>
>
>  for libwebsocket_service(context, VERYSHORTTIMEOUT) otherwise it will
>> take a long time before the external thread request could be handled...
>> This brings me to wonder why it is not possible to loop on
>> libwebsocket_service(context, VERYLONGTIMEOUT) that would be interrupted
>> by our external thread instead? Thus lws thread is asleeped most of the
>> time when there is no incoming IP traffic but can be awaken as soon as
>> some outgoing traffic needs to be sent (generated from the external
>> thread).
>>
>
> Well lws is a singlethreaded nonblocking implementation.  It is designed
> to run on small resource-costrained platforms at the same time a being able
> to service huge numbers of connections in a lightweight way.  It does not
> require threads.
>
> Poll can only wait on file or socket descriptors... you could reserve a
> socket to signal to break out and reload the poll() because somebody's
> attribute changed.  But in most real usage scenarios, some other connection
> is always wanting service before the timeout anyway and as more connections
> are seen the less it matters about latency from waiting for poll() reload.
>
> -Andy
>
>  I hope my two questions are clear..
>>
>> Thanks in advance for your rely.
>>
>> Best regards, Thomas
>>
>> 2013/12/18 "Andy Green (林安廸)" <andy at warmcat.com
>> <mailto:andy at warmcat.com>>
>>
>>
>>     On 18/12/13 06:40, the mail apparently from Andy Green included:
>>
>>
>>
>>         Eugene Agafonov <e.a.agafonov at gmail.com
>>         <mailto:e.a.agafonov at gmail.com>> wrote:
>>
>>             Hi!
>>
>>             I've just implemented the similar scenario just like it is
>>             stated
>>             in document
>>             http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__
>> tree/README.coding
>>
>>             <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/
>> tree/README.coding>
>>
>>
>>
>>     I have a mutex-protected queue of outgoing events (as a std::list of
>>
>>             strings). Each string is intended to be send in separated
>>             message.
>>
>>             The writer thread posts data onto queue and calls
>>             libwebsocket_callback_on___writable(context, wsi)
>>
>>
>>             Protocol callback gets LWS_CALLBACK_SERVER_WRITEABLE as soon
>> as
>>             websocket thread is ready to send data. Once it comes, the
>>             callback
>>             extracts strings from the queue one by one and writes them to
>>             socket using libwebsocket_write.
>>
>>             I'm quite newby in writing WS server with libwebsockets. Any
>>             comments from WS gurus are welcome
>>
>>
>>         What you're doing is a good way to interact with what lws needs
>> and
>>         get what you want from threading.
>>
>>         There's one danger with it, but that is a very tightly defined
>> race
>>         rather than corruption or whatever.
>>
>>
>>     Well... there's another thing to take care about... you need to
>>     manage your private list of live wsis carefully.  It means managing
>>     that list (with any of your locking needed) at the ESTABLISHED /
>>     CLOSED callback for the WSI.  That should be enough I think.
>>
>>
>>         The issue is that the 'wait for writeable' is oneshot, so after it
>>         becomes true in the poll lws will clear the POLLOUT wait flag.
>>  But
>>         because the threads are asynchronous, eventually the other thread
>>         will come at the wrong time and set it just as we're clearing it
>>         with
>>         indeterminate results.  Because it's read-modify-write and flow
>>         control code also modifies this, there's a low probability
>> deadlock
>>         case where POLLIN or POLLOUT from one thread or another did not
>> get
>>         set for wait as the thread thinks it is.
>>
>>         If the extpoll stuff is used the issue can be worse depending on
>>         what
>>         it is doing.
>>
>>         However the approach you're taking is otherwise really workable I
>>         think.  To make it rock solid you'll need to patch lws to
>>         mutex-protect the few places it changes poll wait flags.
>>
>>
>>     I added a patch
>>
>>     http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__
>> commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e
>>
>>     <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=
>> 7a1327977ac10bdbace0012274b8ae889219880e>
>>
>>     that lets you do this in a lock and unlock callback in the user
>>     code, in one place.
>>
>>     If you're not using threading you don't need to do anything new.
>>
>>     -Andy
>>
>>
>>         -Andy
>>
>>             BR, Eugene Agafonov.
>>
>>             On Tuesday 17 December 2013 12:03:43 Thomas Spitz wrote:
>>
>>                 Hello everyone,
>>
>>                 I am trying to find a way for external threads to ask
>>                 libwebsocket
>>
>>             server
>>
>>                 to send websocket message on their behalf (as they
>>                 cannot do it
>>
>>             directly).
>>
>>                 The only way I see at the present is to make a loop with
>>                 libwebsocket_service (context, VERYSHORTTIME); and test
>>                 potential
>>
>>             incoming
>>
>>                 messages from other threads. This is not very elegant as
>> CPU
>>                 computes
>>
>>             all
>>
>>                 the time and not very reliable as I should put
>>                 VERYSHORTTIME at 0
>>                 if
>>
>>             I
>>
>>                 don't want to miss any message.
>>
>>                 I tried the following modification in test-server.c :
>>
>>                 void *connexionServeur(void *arg) {
>>
>>                     struct libwebsocket_context *context = arg; while (1)
>> {
>>                     if(libwebsocket_service(__context, 60000)<0){
>>
>>
>>                     printf("Problem!\n");
>>
>>                     break; } }
>>
>>                     return NULL ;
>>
>>                     } int statut_Thread; pthread_t Thread; statut_Thread =
>>                     pthread_create(&Thread, NULL, connexionServeur,
>>
>>             context
>>
>>                     );
>>
>>                     if (statut_Thread != 0) {
>>
>>                     printf("Problem with thread creation \n"); } while
>>                     (n >= 0 &&
>>                     !force_exit) { struct timeval tv; gettimeofday(&tv,
>>                     NULL); if
>>                     (((unsigned int)tv.tv_usec - oldus) > 50000) {
>>
>>
>>             libwebsocket_callback_on___writable_all_protocol(&__
>> protocols[PROTOCOL_DUMB_I
>>
>>
>>
>>
>>     NCREMENT]); oldus = tv.tv_usec;
>>
>>                     libwebsocket_service(context, 0);
>>
>>                     }
>>
>>                     usleep(10000); }
>>
>>
>>                 but it crashes quickly as soon as I try drawing . I
>>                 suppose that
>>                 libwebsocket_service is called twice at the same time and
>> it
>>                 creates conflicts that lead to stop crash the websocket.
>>
>>                 I have seen in the mailing list that Thomas Koeper has
>>                 already
>>                 had a similar question
>>
>>             http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-_
>> _April/000403.html
>>
>>             <http://ml.libwebsockets.org/pipermail/libwebsockets/2013-
>> April/000403.html>
>>
>>
>>
>>     . At that time, he was thinking of creating a mutex on
>>
>>                 libwebsocket_service. Do you think this is the right
>>                 solution?
>>
>>                 Thanks in advance for your help Best regards, Thomas
>>
>>             _________________________________________________
>> Libwebsockets
>>             mailing list Libwebsockets at ml.__libwebsockets.org
>>             <mailto:Libwebsockets at ml.libwebsockets.org>
>>             http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>             <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>
>>
>>
>>         _________________________________________________ Libwebsockets
>>         mailing
>>         list Libwebsockets at ml.__libwebsockets.org
>>         <mailto:Libwebsockets at ml.libwebsockets.org>
>>         http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>
>>
>>
>>     _________________________________________________
>>     Libwebsockets mailing list
>>     Libwebsockets at ml.__libwebsockets.org
>>     <mailto:Libwebsockets at ml.libwebsockets.org>
>>     http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>     <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20131220/df304a60/attachment-0001.html>


More information about the Libwebsockets mailing list