[Libwebsockets] Interaction of external threads with libwebsockets server

Thomas Spitz thomas.spitz at hestia-france.com
Wed Dec 25 13:14:30 CET 2013


>
> The choices seem to boil down to this kind of "add a fake descriptor"
> thing (although everything, including the "interrupt the poll" descriptor
> and the use of it should be defined inside the library), or maybe change to
> use ppoll() and fire signals at it.

ppoll() could be an interesting solution but it only interrupt the poll. I
was thinking of interrupting poll() using a named pipe in which I would
have told lws which wsi it needs to write to. The complete process would
have been the following:
1) Before libwebsocket_create_context(), I create the named pipe.
2) For every client connection, I book for a shm
in LWS_CALLBACK_ESTABLISHED through which I will share incoming data with
my main thread
3) My main thread process the incoming data and store the answer into the
shm. It then indicates lws that an answer is ready for a given wsi
indicating the ID of the shm in the named pipe
4) lws poll() is interupted and it knows immediatly which wsi it needs to
write to thanks to the ID of the shm. If the wsi is closed in the
meanwhile, lws indicate it to the shm in LWS_CALLBACK_CLOSED

If I use ppoll(), I could keep almost the same principle but I would then
need to add a SIGUSR1 and a handler OR loop through my client shm array
each time ppoll() got interupted with EINTR flag set... Finally I am still
wondering whether my solution is not simpler?

Either way lws_change_pollfd() is central to the solution.

With my solution or even ppoll one, I don't see when I need to call
lws_change_pollfd()
especially as lws_change_pollfd needs a pointer to wsi which I cannot give
as my interrupt concerns the complete context and not a special wsi...? I
must miss a point.

If you pick a signal like SIGUSR1 and install a do-nothing handler for it,
> firing SIGUSR1 at the process from itself in lws_change_pollfd() and using
> ppoll() could be a really small and robust solution.
> Since the signal is handled it doesn't do anything except interrupt the
> ppoll causing a pollfd reload.
> You only need to fire the signal the first time anything wants to
> interrupt the wait *from another thread* (because if the lws thread is in
> poll(), it isn't doing anything else).  If a pollfd raced it and changed
> first, there's no problem with an additional signal interrupting the next
> ppoll loop.

Ideally, if it is not too much to ask, a simple example of code would be
ideal.

BR, Thomas



2013/12/25 "Andy Green (林安廸)" <andy at warmcat.com>

> On 25/12/13 02:06, the mail apparently from Thomas Spitz included:
>
>  Dear Andy,
>>
>> In accordance to your advice, I have created a patch to signal lws to
>> break out and reload the poll() when an external thread needs to
>> interact with lws. This is done by sending a filedescriptor to lws
>> on libwebsocket_create_context().
>>
>> I am novice with lws so please let me know whether it is correct.
>>
>
> Thanks... it's not incorrect, but I'm wondering if this is the best way.
>
> The choices seem to boil down to this kind of "add a fake descriptor"
> thing (although everything, including the "interrupt the poll" descriptor
> and the use of it should be defined inside the library), or maybe change to
> use ppoll() and fire signals at it.
>
> Either way lws_change_pollfd() is central to the solution.
>
> ppoll() is the same as poll() but it lets you basically choose signals to
> wait on inside the poll loop as well as the fds.
>
> If you pick a signal like SIGUSR1 and install a do-nothing handler for it,
> firing SIGUSR1 at the process from itself in lws_change_pollfd() and using
> ppoll() could be a really small and robust solution.
>
> Since the signal is handled it doesn't do anything except interrupt the
> ppoll causing a pollfd reload.
>
> You only need to fire the signal the first time anything wants to
> interrupt the wait *from another thread* (because if the lws thread is in
> poll(), it isn't doing anything else).  If a pollfd raced it and changed
> first, there's no problem with an additional signal interrupting the next
> ppoll loop.
>
> -Andy
>
>  Have a nice christmas,
>>
>> Best regards,
>> Thomas
>>
>>
>> 2013/12/21 "Andy Green (林安廸)" <andy at warmcat.com
>> <mailto:andy at warmcat.com>>
>>
>>
>>     On 21/12/13 10:49, the mail apparently from "Andy Green (林安廸)"
>>     included:
>>
>>         On 20/12/13 19:38, the mail apparently from Thomas Spitz included:
>>
>>             Hello Andy,
>>
>>             In fact, in my application, I mostly often only have 0 to 5
>>             active
>>             connections at the same time with a maximum number of
>>             simultaneous
>>             connections restricted to 32 (Besides, why is there no
>>             simple solution
>>             to reduce the maximum simultaneous connection? I have seen
>>             that we need
>>             to reduce the max system fds...).
>>
>>
>>         You should be able to do
>>
>>         ulimit -n 64
>>
>>         before starting the server.  It only affects that shell and
>>         children. If
>>         you're bothered by the memory footprint of the parent shell, it
>>         should
>>         be workable to do an exec instead of running as a child.
>>
>>         The default is 1024 on most systems... the way we deal with fd
>>         lookups
>>         it's just not that expensive to allow 1024 (a few KB).
>>
>>             Therefore, I think I should implement your following
>>             proposal using
>>             External_Poll I guess:
>>
>>                  Poll can only wait on file or socket descriptors... you
>>             could
>>                  reserve a socket to signal to break out and reload the
>>             poll()
>>                  because somebody's attribute changed.
>>
>>
>>         Actually I was imagining adding it in lws.
>>
>>
>>     I just pushed a patch
>>
>>     http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__
>> commit/?id=__91f19d8d799fdce0459c80357b3993__0b3ac55a9c
>>
>>     <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=
>> 91f19d8d799fdce0459c80357b39930b3ac55a9c>
>>
>>     which should give you a single place to signal all pollfd changes
>>     from if you want to have a go at a patch to implement this on lws
>> side.
>>
>>     If you make a new socket or other descriptor-based thing in the
>>     context create code that is open for the lifetime of the client or
>>     server, and add it to the pollfs there with POLLIN event set,
>>     writing a byte to it from lws_change_pollfd() the first time
>>     something changed before the outer loop reloads should do what you
>> want.
>>
>>     -Andy
>>
>>             I tried to do it but my thread is still not able to break
>>             out the
>>             poll(). In my test below (using test-server.c as skeletton),
>>               Although I
>>
>>
>>         I think it must mean your test is broken.  Otherwise you're
>>         saying that
>>         poll() doesn't work.
>>
>>         -Andy
>>
>>             write in the tube that is monitored by poll I see that my
>>             asyncSending
>>             is never able to break up poll(). poll() always break up on
>>             10s timeout
>>             whereas my thread write into the tube every second.
>>
>>             int tube[2];
>>
>>             (...)
>>
>>             void *asyncSending(void *arg) {
>>
>>             unsigned char c=0;
>>             printf("START asynchronous sending\n");
>>
>>             while(1){
>>             c++;
>>             printf("write in tube\n");
>>             if(write(tube[1],&c,1)!=1){
>>             printf("Erreur write %s\n", strerror(errno));
>>             }
>>             //libwebsocket_callback_on___writable_all_protocol(&__
>> protocols[PROTOCOL_DUMB___INCREMENT]);
>>
>>
>>             }
>>
>>             printf("END asynchronous sending\n");
>>             return NULL ;
>>
>>             }
>>
>>
>>                int main(int argc, char **argv) {
>>
>>                       (...)
>>
>>             n = 0;
>>
>>             printf("Tube creation\n");
>>             if(pipe(tube)!=0){
>>             printf("Pipe creation error %s\n", strerror(errno));
>>             }
>>
>>             if (count_pollfds >= max_poll_elements) {
>>             lwsl_err("LWS_CALLBACK_ADD___POLL_FD: too many sockets to
>>
>>             track\n");
>>             return 1;
>>             }
>>
>>             fd_lookup[tube[0]] = count_pollfds;
>>             pollfds[count_pollfds].fd = fd_lookup[tube[0]];
>>             pollfds[count_pollfds].events = POLLIN | POLLPRI |POLL_OUT;
>>             pollfds[count_pollfds++].__revents = 0;
>>
>>
>>             int statut_Thread;
>>             pthread_t Thread;
>>
>>             statut_Thread = pthread_create(&Thread, NULL, asyncSending,
>>             context);
>>             if (statut_Thread != 0) {
>>             printf("Thread creation errror \n");
>>             }
>>
>>             while (n >= 0 && !force_exit) {
>>
>>             /*
>>             * this represents an existing server's single poll action
>>             * which also includes libwebsocket sockets
>>             */
>>
>>             n = poll(pollfds, count_pollfds, 10000);//10s
>>             if (n < 0)
>>             continue;
>>
>>             if (n)
>>             for (n = 0; n < count_pollfds; n++)
>>             if (pollfds[n].revents)
>>             /*
>>             * returns immediately if the fd does not
>>             * match anything under libwebsockets
>>             * control
>>             */
>>             if (libwebsocket_service_fd(__context,
>>
>>             &pollfds[n]) < 0)
>>             goto done;
>>
>>                              (...)
>>                      }
>>             }
>>
>>             Thanks again in advance for your help.
>>
>>             BR, Thomas
>>
>>
>>             2013/12/19 "Andy Green (林安廸)" <andy at warmcat.com
>>             <mailto:andy at warmcat.com>
>>             <mailto:andy at warmcat.com <mailto:andy at warmcat.com>>>
>>
>>                  On 19/12/13 02:12, the mail apparently from Thomas
>>             Spitz included:
>>
>>                      Hello Andy, hello Eugene,
>>
>>                      If I understood it correctly, thanks to your patch,
>>             we can now
>>                      prevent
>>                      our external thread(s) from calling
>>                      libwebsocket_callback_on_____writable
>>
>>                      while lws is actively dealing with POLLIN and
>>             POLLOUT of fds
>>                      using our
>>                      own mutex? Thus we are sure that the request(s)
>>             from our external
>>                      thread(s) will not be lost, or at least trigger
>>                      a LWS_CALLBACK_SERVER_WRITEABLE?
>>
>>
>>                  I'm not sure it won't get lost, but it won't leave
>>             things in an
>>                  indeterminate state.  It can still happen the
>>             non-service thread
>>                  asks to set POLLOUT just as we're about to take the
>>             mutex and
>>             clear it.
>>
>>                  At the moment that'll still break at low probability
>>             but it should
>>                  be solveable.
>>
>>
>>                      My second question is related to performance:
>>
>>                      In the documentation it is said
>>
>>                           If you want to send something, do not just
>>             send it but
>>                      request a
>>                           callback |when the socket is writeable |
>>
>>                      That is why our external thread call
>>                      libwebsocket_callback_on_____writable
>>
>>                      (). Then it is written
>>
>>                           |Usually you will get called back immediately
>>             next time
>>                      around the
>>                           service loop|
>>
>>                      which seems to mean that  (according to my tests)
>>             we have to
>>                      wait libwebsocket_service() to end up before the
>>             socket is
>>             writeable
>>                      again. That implies to set a very short timeout
>>
>>
>>                  Well... you are right if we have one connection.  But
>>             if you have 50
>>                  or 50,000 connections, somebody almost always has
>>             something going on
>>                  before the timeout.
>>
>>                  Also in your user callback, you can take the approach
>>             like in the
>>                  test apps to loop using lws_send_pipe_choked() test to
>>             absolutely
>>                  stuff your pipe each time you get service.  That
>>             doesn't delay
>>                  anything looping there because it only loops while the
>>             send won't
>>                  block.  And it means the kernel is passing out what you
>>             stuffed into
>>                  the pipe while you wait through any latency in the next
>>             "writeable"
>>                  service.
>>
>>
>>                      for libwebsocket_service(context, VERYSHORTTIMEOUT)
>>             otherwise it
>>                      will
>>                      take a long time before the external thread request
>>             could be
>>                      handled...
>>                      This brings me to wonder why it is not possible to
>>             loop on
>>                      libwebsocket_service(context, VERYLONGTIMEOUT) that
>>             would be
>>                      interrupted
>>                      by our external thread instead? Thus lws thread is
>>             asleeped most
>>                      of the
>>                      time when there is no incoming IP traffic but can
>>             be awaken as
>>                      soon as
>>                      some outgoing traffic needs to be sent (generated
>>             from the
>>                      external thread).
>>
>>
>>                  Well lws is a singlethreaded nonblocking
>>             implementation.  It is
>>                  designed to run on small resource-costrained platforms
>>             at the same
>>                  time a being able to service huge numbers of
>>             connections in a
>>                  lightweight way.  It does not require threads.
>>
>>                  Poll can only wait on file or socket descriptors... you
>>             could
>>                  reserve a socket to signal to break out and reload the
>>             poll()
>>                  because somebody's attribute changed.  But in most real
>>             usage
>>                  scenarios, some other connection is always wanting
>>             service before
>>                  the timeout anyway and as more connections are seen the
>>             less it
>>                  matters about latency from waiting for poll() reload.
>>
>>                  -Andy
>>
>>                      I hope my two questions are clear..
>>
>>                      Thanks in advance for your rely.
>>
>>                      Best regards, Thomas
>>
>>                      2013/12/18 "Andy Green (林安廸)" <andy at warmcat.com
>>             <mailto:andy at warmcat.com>
>>                      <mailto:andy at warmcat.com <mailto:andy at warmcat.com>>
>>                      <mailto:andy at warmcat.com <mailto:andy at warmcat.com>
>>             <mailto:andy at warmcat.com <mailto:andy at warmcat.com>>>>
>>
>>
>>                           On 18/12/13 06:40, the mail apparently from
>>             Andy Green
>>                      included:
>>
>>
>>
>>                               Eugene Agafonov <e.a.agafonov at gmail.com
>>             <mailto:e.a.agafonov at gmail.com>
>>                      <mailto:e.a.agafonov at gmail.com
>>             <mailto:e.a.agafonov at gmail.com>__>
>>                               <mailto:e.a.agafonov at gmail.com
>>             <mailto:e.a.agafonov at gmail.com>
>>                      <mailto:e.a.agafonov at gmail.com
>>             <mailto:e.a.agafonov at gmail.com>__>__>> wrote:
>>
>>                                   Hi!
>>
>>                                   I've just implemented the similar
>>             scenario just
>>                      like it is
>>                                   stated
>>                                   in document
>>
>>             http://git.libwebsockets.org/______cgi-bin/cgit/
>> libwebsockets/______tree/README.coding
>>             <http://git.libwebsockets.org/____cgi-bin/cgit/
>> libwebsockets/____tree/README.coding>
>>
>>
>>
>>             <http://git.libwebsockets.org/____cgi-bin/cgit/
>> libwebsockets/____tree/README.coding
>>             <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>> _tree/README.coding>>
>>
>>
>>
>>
>>             <http://git.libwebsockets.org/____cgi-bin/cgit/
>> libwebsockets/____tree/README.coding
>>             <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>> _tree/README.coding>
>>
>>
>>             <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>> _tree/README.coding
>>             <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/
>> tree/README.coding>>>
>>
>>
>>
>>
>>                           I have a mutex-protected queue of outgoing
>>             events (as a
>>                      std::list of
>>
>>                                   strings). Each string is intended to
>>             be send in
>>                      separated
>>                                   message.
>>
>>                                   The writer thread posts data onto
>>             queue and calls
>>
>>               libwebsocket_callback_on_______writable(context, wsi)
>>
>>
>>
>>                                   Protocol callback gets
>>                      LWS_CALLBACK_SERVER_WRITEABLE as soon as
>>                                   websocket thread is ready to send
>>             data. Once it
>>                      comes, the
>>                                   callback
>>                                   extracts strings from the queue one by
>>             one and
>>                      writes them to
>>                                   socket using libwebsocket_write.
>>
>>                                   I'm quite newby in writing WS server
>> with
>>                      libwebsockets. Any
>>                                   comments from WS gurus are welcome
>>
>>
>>                               What you're doing is a good way to
>>             interact with what
>>                      lws needs and
>>                               get what you want from threading.
>>
>>                               There's one danger with it, but that is a
>>             very tightly
>>                      defined race
>>                               rather than corruption or whatever.
>>
>>
>>                           Well... there's another thing to take care
>>             about... you
>>             need to
>>                           manage your private list of live wsis
>>             carefully.  It means
>>                      managing
>>                           that list (with any of your locking needed) at
>> the
>>                      ESTABLISHED /
>>                           CLOSED callback for the WSI.  That should be
>>             enough I think.
>>
>>
>>                               The issue is that the 'wait for writeable'
>>             is oneshot,
>>                      so after it
>>                               becomes true in the poll lws will clear
>>             the POLLOUT
>>                      wait flag.  But
>>                               because the threads are asynchronous,
>>             eventually the
>>                      other thread
>>                               will come at the wrong time and set it
>>             just as we're
>>                      clearing it
>>                               with
>>                               indeterminate results.  Because it's
>>             read-modify-write
>>                      and flow
>>                               control code also modifies this, there's a
>> low
>>                      probability deadlock
>>                               case where POLLIN or POLLOUT from one
>>             thread or another
>>                      did not get
>>                               set for wait as the thread thinks it is.
>>
>>                               If the extpoll stuff is used the issue can
>>             be worse
>>                      depending on
>>                               what
>>                               it is doing.
>>
>>                               However the approach you're taking is
>>             otherwise really
>>                      workable I
>>                               think.  To make it rock solid you'll need
>>             to patch
>>             lws to
>>                               mutex-protect the few places it changes
>>             poll wait flags.
>>
>>
>>                           I added a patch
>>
>>
>>             http://git.libwebsockets.org/______cgi-bin/cgit/
>> libwebsockets/______commit/?id=______7a1327977ac10bdbace0012274b8ae
>> ______889219880e
>>             <http://git.libwebsockets.org/____cgi-bin/cgit/
>> libwebsockets/____commit/?id=____7a1327977ac10bdbace0012274b8ae
>> ____889219880e>
>>
>>
>>
>>             <http://git.libwebsockets.org/____cgi-bin/cgit/
>> libwebsockets/____commit/?id=____7a1327977ac10bdbace0012274b8ae
>> ____889219880e
>>             <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>> _commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e>>
>>
>>
>>
>>
>>             <http://git.libwebsockets.org/____cgi-bin/cgit/
>> libwebsockets/____commit/?id=____7a1327977ac10bdbace0012274b8ae
>> ____889219880e
>>             <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>> _commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e>
>>
>>
>>             <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>> _commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e
>>             <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/
>> commit/?id=7a1327977ac10bdbace0012274b8ae889219880e>>>
>>
>>
>>                           that lets you do this in a lock and unlock
>>             callback in
>>             the user
>>                           code, in one place.
>>
>>                           If you're not using threading you don't need
>>             to do anything
>>                      new.
>>
>>                           -Andy
>>
>>
>>                               -Andy
>>
>>                                   BR, Eugene Agafonov.
>>
>>                                   On Tuesday 17 December 2013 12:03:43
>>             Thomas Spitz
>>                      wrote:
>>
>>                                       Hello everyone,
>>
>>                                       I am trying to find a way for
>>             external threads
>>                      to ask
>>                                       libwebsocket
>>
>>                                   server
>>
>>                                       to send websocket message on their
>>             behalf (as
>>             they
>>                                       cannot do it
>>
>>                                   directly).
>>
>>                                       The only way I see at the present
>>             is to make a
>>                      loop with
>>                                       libwebsocket_service (context,
>>             VERYSHORTTIME);
>>                      and test
>>                                       potential
>>
>>                                   incoming
>>
>>                                       messages from other threads. This
>>             is not very
>>                      elegant as CPU
>>                                       computes
>>
>>                                   all
>>
>>                                       the time and not very reliable as
>>             I should put
>>                                       VERYSHORTTIME at 0
>>                                       if
>>
>>                                   I
>>
>>                                       don't want to miss any message.
>>
>>                                       I tried the following modification
>> in
>>                      test-server.c :
>>
>>                                       void *connexionServeur(void *arg) {
>>
>>                                           struct libwebsocket_context
>>             *context = arg;
>>                      while (1) {
>>
>>               if(libwebsocket_service(______context,
>>
>>             60000)<0){
>>
>>
>>                                           printf("Problem!\n");
>>
>>                                           break; } }
>>
>>                                           return NULL ;
>>
>>                                           } int statut_Thread; pthread_t
>>             Thread;
>>                      statut_Thread =
>>                                           pthread_create(&Thread, NULL,
>>             connexionServeur,
>>
>>                                   context
>>
>>                                           );
>>
>>                                           if (statut_Thread != 0) {
>>
>>                                           printf("Problem with thread
>>             creation \n");
>>                      } while
>>                                           (n >= 0 &&
>>                                           !force_exit) { struct timeval
>> tv;
>>                      gettimeofday(&tv,
>>                                           NULL); if
>>                                           (((unsigned int)tv.tv_usec -
>>             oldus) >
>>             50000) {
>>
>>
>>
>>
>>             libwebsocket_callback_on_______writable_all_protocol(&______
>> protocols[PROTOCOL_DUMB_I
>>
>>
>>
>>
>>
>>
>>                           NCREMENT]); oldus = tv.tv_usec;
>>
>>                                           libwebsocket_service(context,
>> 0);
>>
>>                                           }
>>
>>                                           usleep(10000); }
>>
>>
>>                                       but it crashes quickly as soon as
>>             I try
>>             drawing . I
>>                                       suppose that
>>                                       libwebsocket_service is called
>>             twice at the
>>                      same time and it
>>                                       creates conflicts that lead to
>>             stop crash the
>>                      websocket.
>>
>>                                       I have seen in the mailing list
>>             that Thomas
>>                      Koeper has
>>                                       already
>>                                       had a similar question
>>
>>
>>             http://ml.libwebsockets.org/______pipermail/libwebsockets/__
>> 2013-____April/000403.html
>>             <http://ml.libwebsockets.org/____pipermail/libwebsockets/
>> 2013-____April/000403.html>
>>
>>
>>
>>             <http://ml.libwebsockets.org/____pipermail/libwebsockets/
>> 2013-____April/000403.html
>>             <http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-
>> __April/000403.html>>
>>
>>
>>
>>
>>             <http://ml.libwebsockets.org/____pipermail/libwebsockets/
>> 2013-____April/000403.html
>>             <http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-
>> __April/000403.html>
>>
>>
>>             <http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-
>> __April/000403.html
>>             <http://ml.libwebsockets.org/pipermail/libwebsockets/2013-
>> April/000403.html>>>
>>
>>
>>
>>
>>                           . At that time, he was thinking of creating a
>>             mutex on
>>
>>                                       libwebsocket_service. Do you think
>>             this is the
>>                      right
>>                                       solution?
>>
>>                                       Thanks in advance for your help
>>             Best regards,
>>                      Thomas
>>
>>
>>               _____________________________________________________
>>                      Libwebsockets
>>                                   mailing list
>>             Libwebsockets at ml.__libwebsocke____ts.org
>>             <http://libwebsocke__ts.org>
>>                      <http://libwebsockets.org>
>>                                   <mailto:Libwebsockets at ml.
>>             <mailto:Libwebsockets at ml.>__lib__websockets.org
>>
>>             <http://libwebsockets.org>
>>                      <mailto:Libwebsockets at ml.__libwebsockets.org
>>             <mailto:Libwebsockets at ml.libwebsockets.org>>>
>>             http://ml.libwebsockets.org/______mailman/listinfo/__
>> libwebsockets
>>             <http://ml.libwebsockets.org/____mailman/listinfo/
>> libwebsockets>
>>
>>             <http://ml.libwebsockets.org/____mailman/listinfo/__
>> libwebsockets
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>> >>
>>
>>
>>             <http://ml.libwebsockets.org/____mailman/listinfo/__
>> libwebsockets
>>
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>> >
>>
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets<
>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>__>
>>
>>
>>
>>               _____________________________________________________
>>                      Libwebsockets
>>                               mailing
>>                               list
>>             Libwebsockets at ml.__libwebsocke____ts.org
>>             <http://libwebsocke__ts.org>
>>                      <http://libwebsockets.org>
>>                               <mailto:Libwebsockets at ml.
>>             <mailto:Libwebsockets at ml.>__lib__websockets.org
>>
>>             <http://libwebsockets.org>
>>                      <mailto:Libwebsockets at ml.__libwebsockets.org
>>             <mailto:Libwebsockets at ml.libwebsockets.org>>>
>>             http://ml.libwebsockets.org/______mailman/listinfo/__
>> libwebsockets
>>             <http://ml.libwebsockets.org/____mailman/listinfo/
>> libwebsockets>
>>
>>             <http://ml.libwebsockets.org/____mailman/listinfo/__
>> libwebsockets
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>> >>
>>
>>
>>             <http://ml.libwebsockets.org/____mailman/listinfo/__
>> libwebsockets
>>
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>> >
>>
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets<
>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>__>
>>
>>
>>
>>               _____________________________________________________
>>                           Libwebsockets mailing list
>>                           Libwebsockets at ml.__libwebsocke____ts.org
>>             <http://libwebsocke__ts.org>
>>                      <http://libwebsockets.org>
>>                           <mailto:Libwebsockets at ml.
>>             <mailto:Libwebsockets at ml.>__lib__websockets.org
>>
>>             <http://libwebsockets.org>
>>                      <mailto:Libwebsockets at ml.__libwebsockets.org
>>             <mailto:Libwebsockets at ml.libwebsockets.org>>>
>>             http://ml.libwebsockets.org/______mailman/listinfo/__
>> libwebsockets
>>             <http://ml.libwebsockets.org/____mailman/listinfo/
>> libwebsockets>
>>
>>             <http://ml.libwebsockets.org/____mailman/listinfo/__
>> libwebsockets
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>> >>
>>
>>
>>             <http://ml.libwebsockets.org/____mailman/listinfo/__
>> libwebsockets
>>
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>> >
>>
>>             <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets<
>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>__>
>>
>>
>>
>>
>>         _________________________________________________
>>         Libwebsockets mailing list
>>         Libwebsockets at ml.__libwebsockets.org
>>         <mailto:Libwebsockets at ml.libwebsockets.org>
>>         http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20131225/076e5226/attachment-0001.html>


More information about the Libwebsockets mailing list