[Libwebsockets] Interaction of external threads with libwebsockets server

Thomas Spitz thomas.spitz at hestia-france.com
Mon Dec 23 15:40:28 CET 2013


Hello Andy,

You should be able to do
> ulimit -n 64
> before starting the server.  It only affects that shell and children. If
> you're bothered by the memory footprint of the parent shell, it should
> be workable to do an exec instead of running as a child.

I wasn't hesitating to launch libwebsockets in a separate processus from my
main application but your solution for limiting max simultenous connection
will force me not to launch libwebsockets server in a separate thread but
in a separate processus.

Actually I was imagining adding it in lws.

I'm really sorry but I don't see how to add my external thread notification
inside libwebsockets. Should I put it at the end
of libwebsocket_create_context() in libwebsockets.c? If so, it means I have
my own customized lib... I also tried to access context->fds,
context->fds_count from test-server.c but it is not allowed... Otherwise,
just for info, at the end I would prefer using IPC message queues instead
of pipe... I hope it will not create problems with poll()...

I hope you could push me a bit further again..

Thanks in advance

Best regards,
Thomas


2013/12/21 "Andy Green (林安廸)" <andy at warmcat.com>

> On 21/12/13 10:49, the mail apparently from "Andy Green (林安廸)" included:
>
>  On 20/12/13 19:38, the mail apparently from Thomas Spitz included:
>>
>>> Hello Andy,
>>>
>>> In fact, in my application, I mostly often only have 0 to 5 active
>>> connections at the same time with a maximum number of simultaneous
>>> connections restricted to 32 (Besides, why is there no simple solution
>>> to reduce the maximum simultaneous connection? I have seen that we need
>>> to reduce the max system fds...).
>>>
>>
>> You should be able to do
>>
>> ulimit -n 64
>>
>> before starting the server.  It only affects that shell and children. If
>> you're bothered by the memory footprint of the parent shell, it should
>> be workable to do an exec instead of running as a child.
>>
>> The default is 1024 on most systems... the way we deal with fd lookups
>> it's just not that expensive to allow 1024 (a few KB).
>>
>>  Therefore, I think I should implement your following proposal using
>>> External_Poll I guess:
>>>
>>>     Poll can only wait on file or socket descriptors... you could
>>>     reserve a socket to signal to break out and reload the poll()
>>>     because somebody's attribute changed.
>>>
>>
>> Actually I was imagining adding it in lws.
>>
>
> I just pushed a patch
>
> http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=
> 91f19d8d799fdce0459c80357b39930b3ac55a9c
>
> which should give you a single place to signal all pollfd changes from if
> you want to have a go at a patch to implement this on lws side.
>
> If you make a new socket or other descriptor-based thing in the context
> create code that is open for the lifetime of the client or server, and add
> it to the pollfs there with POLLIN event set, writing a byte to it from
> lws_change_pollfd() the first time something changed before the outer loop
> reloads should do what you want.
>
> -Andy
>
>  I tried to do it but my thread is still not able to break out the
>>> poll(). In my test below (using test-server.c as skeletton),  Although I
>>>
>>
>> I think it must mean your test is broken.  Otherwise you're saying that
>> poll() doesn't work.
>>
>> -Andy
>>
>>  write in the tube that is monitored by poll I see that my asyncSending
>>> is never able to break up poll(). poll() always break up on 10s timeout
>>> whereas my thread write into the tube every second.
>>>
>>> int tube[2];
>>>
>>> (...)
>>>
>>> void *asyncSending(void *arg) {
>>>
>>> unsigned char c=0;
>>> printf("START asynchronous sending\n");
>>>
>>> while(1){
>>> c++;
>>> printf("write in tube\n");
>>> if(write(tube[1],&c,1)!=1){
>>> printf("Erreur write %s\n", strerror(errno));
>>> }
>>> //libwebsocket_callback_on_writable_all_protocol(&
>>> protocols[PROTOCOL_DUMB_INCREMENT]);
>>>
>>> }
>>>
>>> printf("END asynchronous sending\n");
>>> return NULL ;
>>>
>>> }
>>>
>>>
>>>   int main(int argc, char **argv) {
>>>
>>>          (...)
>>>
>>> n = 0;
>>>
>>> printf("Tube creation\n");
>>> if(pipe(tube)!=0){
>>> printf("Pipe creation error %s\n", strerror(errno));
>>> }
>>>
>>> if (count_pollfds >= max_poll_elements) {
>>> lwsl_err("LWS_CALLBACK_ADD_POLL_FD: too many sockets to track\n");
>>> return 1;
>>> }
>>>
>>> fd_lookup[tube[0]] = count_pollfds;
>>> pollfds[count_pollfds].fd = fd_lookup[tube[0]];
>>> pollfds[count_pollfds].events = POLLIN | POLLPRI |POLL_OUT;
>>> pollfds[count_pollfds++].revents = 0;
>>>
>>> int statut_Thread;
>>> pthread_t Thread;
>>>
>>> statut_Thread = pthread_create(&Thread, NULL, asyncSending, context);
>>> if (statut_Thread != 0) {
>>> printf("Thread creation errror \n");
>>> }
>>>
>>> while (n >= 0 && !force_exit) {
>>>
>>> /*
>>> * this represents an existing server's single poll action
>>> * which also includes libwebsocket sockets
>>> */
>>>
>>> n = poll(pollfds, count_pollfds, 10000);//10s
>>> if (n < 0)
>>> continue;
>>>
>>> if (n)
>>> for (n = 0; n < count_pollfds; n++)
>>> if (pollfds[n].revents)
>>> /*
>>> * returns immediately if the fd does not
>>> * match anything under libwebsockets
>>> * control
>>> */
>>> if (libwebsocket_service_fd(context,
>>> &pollfds[n]) < 0)
>>> goto done;
>>>
>>>                 (...)
>>>         }
>>> }
>>>
>>> Thanks again in advance for your help.
>>>
>>> BR, Thomas
>>>
>>>
>>> 2013/12/19 "Andy Green (林安廸)" <andy at warmcat.com
>>> <mailto:andy at warmcat.com>>
>>>
>>>     On 19/12/13 02:12, the mail apparently from Thomas Spitz included:
>>>
>>>         Hello Andy, hello Eugene,
>>>
>>>         If I understood it correctly, thanks to your patch, we can now
>>>         prevent
>>>         our external thread(s) from calling
>>>         libwebsocket_callback_on___writable
>>>         while lws is actively dealing with POLLIN and POLLOUT of fds
>>>         using our
>>>         own mutex? Thus we are sure that the request(s) from our external
>>>         thread(s) will not be lost, or at least trigger
>>>         a LWS_CALLBACK_SERVER_WRITEABLE?
>>>
>>>
>>>     I'm not sure it won't get lost, but it won't leave things in an
>>>     indeterminate state.  It can still happen the non-service thread
>>>     asks to set POLLOUT just as we're about to take the mutex and
>>> clear it.
>>>
>>>     At the moment that'll still break at low probability but it should
>>>     be solveable.
>>>
>>>
>>>         My second question is related to performance:
>>>
>>>         In the documentation it is said
>>>
>>>              If you want to send something, do not just send it but
>>>         request a
>>>              callback |when the socket is writeable |
>>>
>>>         That is why our external thread call
>>>         libwebsocket_callback_on___writable
>>>         (). Then it is written
>>>
>>>              |Usually you will get called back immediately next time
>>>         around the
>>>              service loop|
>>>
>>>         which seems to mean that  (according to my tests) we have to
>>>         wait libwebsocket_service() to end up before the socket is
>>> writeable
>>>         again. That implies to set a very short timeout
>>>
>>>
>>>     Well... you are right if we have one connection.  But if you have 50
>>>     or 50,000 connections, somebody almost always has something going on
>>>     before the timeout.
>>>
>>>     Also in your user callback, you can take the approach like in the
>>>     test apps to loop using lws_send_pipe_choked() test to absolutely
>>>     stuff your pipe each time you get service.  That doesn't delay
>>>     anything looping there because it only loops while the send won't
>>>     block.  And it means the kernel is passing out what you stuffed into
>>>     the pipe while you wait through any latency in the next "writeable"
>>>     service.
>>>
>>>
>>>         for libwebsocket_service(context, VERYSHORTTIMEOUT) otherwise it
>>>         will
>>>         take a long time before the external thread request could be
>>>         handled...
>>>         This brings me to wonder why it is not possible to loop on
>>>         libwebsocket_service(context, VERYLONGTIMEOUT) that would be
>>>         interrupted
>>>         by our external thread instead? Thus lws thread is asleeped most
>>>         of the
>>>         time when there is no incoming IP traffic but can be awaken as
>>>         soon as
>>>         some outgoing traffic needs to be sent (generated from the
>>>         external thread).
>>>
>>>
>>>     Well lws is a singlethreaded nonblocking implementation.  It is
>>>     designed to run on small resource-costrained platforms at the same
>>>     time a being able to service huge numbers of connections in a
>>>     lightweight way.  It does not require threads.
>>>
>>>     Poll can only wait on file or socket descriptors... you could
>>>     reserve a socket to signal to break out and reload the poll()
>>>     because somebody's attribute changed.  But in most real usage
>>>     scenarios, some other connection is always wanting service before
>>>     the timeout anyway and as more connections are seen the less it
>>>     matters about latency from waiting for poll() reload.
>>>
>>>     -Andy
>>>
>>>         I hope my two questions are clear..
>>>
>>>         Thanks in advance for your rely.
>>>
>>>         Best regards, Thomas
>>>
>>>         2013/12/18 "Andy Green (林安廸)" <andy at warmcat.com
>>>         <mailto:andy at warmcat.com>
>>>         <mailto:andy at warmcat.com <mailto:andy at warmcat.com>>>
>>>
>>>
>>>              On 18/12/13 06:40, the mail apparently from Andy Green
>>>         included:
>>>
>>>
>>>
>>>                  Eugene Agafonov <e.a.agafonov at gmail.com
>>>         <mailto:e.a.agafonov at gmail.com>
>>>                  <mailto:e.a.agafonov at gmail.com
>>>         <mailto:e.a.agafonov at gmail.com>__>> wrote:
>>>
>>>                      Hi!
>>>
>>>                      I've just implemented the similar scenario just
>>>         like it is
>>>                      stated
>>>                      in document
>>>
>>> http://git.libwebsockets.org/____cgi-bin/cgit/libwebsockets/
>>> ____tree/README.coding
>>>
>>>
>>> <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>>> _tree/README.coding>
>>>
>>>
>>>
>>>
>>> <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>>> _tree/README.coding
>>>
>>>
>>> <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/
>>> tree/README.coding>>
>>>
>>>
>>>
>>>
>>>              I have a mutex-protected queue of outgoing events (as a
>>>         std::list of
>>>
>>>                      strings). Each string is intended to be send in
>>>         separated
>>>                      message.
>>>
>>>                      The writer thread posts data onto queue and calls
>>>                      libwebsocket_callback_on_____writable(context, wsi)
>>>
>>>
>>>                      Protocol callback gets
>>>         LWS_CALLBACK_SERVER_WRITEABLE as soon as
>>>                      websocket thread is ready to send data. Once it
>>>         comes, the
>>>                      callback
>>>                      extracts strings from the queue one by one and
>>>         writes them to
>>>                      socket using libwebsocket_write.
>>>
>>>                      I'm quite newby in writing WS server with
>>>         libwebsockets. Any
>>>                      comments from WS gurus are welcome
>>>
>>>
>>>                  What you're doing is a good way to interact with what
>>>         lws needs and
>>>                  get what you want from threading.
>>>
>>>                  There's one danger with it, but that is a very tightly
>>>         defined race
>>>                  rather than corruption or whatever.
>>>
>>>
>>>              Well... there's another thing to take care about... you
>>> need to
>>>              manage your private list of live wsis carefully.  It means
>>>         managing
>>>              that list (with any of your locking needed) at the
>>>         ESTABLISHED /
>>>              CLOSED callback for the WSI.  That should be enough I think.
>>>
>>>
>>>                  The issue is that the 'wait for writeable' is oneshot,
>>>         so after it
>>>                  becomes true in the poll lws will clear the POLLOUT
>>>         wait flag.  But
>>>                  because the threads are asynchronous, eventually the
>>>         other thread
>>>                  will come at the wrong time and set it just as we're
>>>         clearing it
>>>                  with
>>>                  indeterminate results.  Because it's read-modify-write
>>>         and flow
>>>                  control code also modifies this, there's a low
>>>         probability deadlock
>>>                  case where POLLIN or POLLOUT from one thread or another
>>>         did not get
>>>                  set for wait as the thread thinks it is.
>>>
>>>                  If the extpoll stuff is used the issue can be worse
>>>         depending on
>>>                  what
>>>                  it is doing.
>>>
>>>                  However the approach you're taking is otherwise really
>>>         workable I
>>>                  think.  To make it rock solid you'll need to patch
>>> lws to
>>>                  mutex-protect the few places it changes poll wait flags.
>>>
>>>
>>>              I added a patch
>>>
>>>
>>> http://git.libwebsockets.org/____cgi-bin/cgit/libwebsockets/
>>> ____commit/?id=____7a1327977ac10bdbace0012274b8ae____889219880e
>>>
>>>
>>> <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>>> _commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e>
>>>
>>>
>>>
>>>
>>> <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/_
>>> _commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e
>>>
>>>
>>> <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=
>>> 7a1327977ac10bdbace0012274b8ae889219880e>>
>>>
>>>
>>>              that lets you do this in a lock and unlock callback in
>>> the user
>>>              code, in one place.
>>>
>>>              If you're not using threading you don't need to do anything
>>>         new.
>>>
>>>              -Andy
>>>
>>>
>>>                  -Andy
>>>
>>>                      BR, Eugene Agafonov.
>>>
>>>                      On Tuesday 17 December 2013 12:03:43 Thomas Spitz
>>>         wrote:
>>>
>>>                          Hello everyone,
>>>
>>>                          I am trying to find a way for external threads
>>>         to ask
>>>                          libwebsocket
>>>
>>>                      server
>>>
>>>                          to send websocket message on their behalf (as
>>> they
>>>                          cannot do it
>>>
>>>                      directly).
>>>
>>>                          The only way I see at the present is to make a
>>>         loop with
>>>                          libwebsocket_service (context, VERYSHORTTIME);
>>>         and test
>>>                          potential
>>>
>>>                      incoming
>>>
>>>                          messages from other threads. This is not very
>>>         elegant as CPU
>>>                          computes
>>>
>>>                      all
>>>
>>>                          the time and not very reliable as I should put
>>>                          VERYSHORTTIME at 0
>>>                          if
>>>
>>>                      I
>>>
>>>                          don't want to miss any message.
>>>
>>>                          I tried the following modification in
>>>         test-server.c :
>>>
>>>                          void *connexionServeur(void *arg) {
>>>
>>>                              struct libwebsocket_context *context = arg;
>>>         while (1) {
>>>                              if(libwebsocket_service(____context,
>>> 60000)<0){
>>>
>>>
>>>                              printf("Problem!\n");
>>>
>>>                              break; } }
>>>
>>>                              return NULL ;
>>>
>>>                              } int statut_Thread; pthread_t Thread;
>>>         statut_Thread =
>>>                              pthread_create(&Thread, NULL,
>>> connexionServeur,
>>>
>>>                      context
>>>
>>>                              );
>>>
>>>                              if (statut_Thread != 0) {
>>>
>>>                              printf("Problem with thread creation \n");
>>>         } while
>>>                              (n >= 0 &&
>>>                              !force_exit) { struct timeval tv;
>>>         gettimeofday(&tv,
>>>                              NULL); if
>>>                              (((unsigned int)tv.tv_usec - oldus) >
>>> 50000) {
>>>
>>>
>>>
>>>
>>> libwebsocket_callback_on_____writable_all_protocol(&____
>>> protocols[PROTOCOL_DUMB_I
>>>
>>>
>>>
>>>
>>>
>>>              NCREMENT]); oldus = tv.tv_usec;
>>>
>>>                              libwebsocket_service(context, 0);
>>>
>>>                              }
>>>
>>>                              usleep(10000); }
>>>
>>>
>>>                          but it crashes quickly as soon as I try
>>> drawing . I
>>>                          suppose that
>>>                          libwebsocket_service is called twice at the
>>>         same time and it
>>>                          creates conflicts that lead to stop crash the
>>>         websocket.
>>>
>>>                          I have seen in the mailing list that Thomas
>>>         Koeper has
>>>                          already
>>>                          had a similar question
>>>
>>>
>>> http://ml.libwebsockets.org/____pipermail/libwebsockets/
>>> 2013-____April/000403.html
>>>
>>>
>>> <http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-
>>> __April/000403.html>
>>>
>>>
>>>
>>>
>>> <http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-
>>> __April/000403.html
>>>
>>>
>>> <http://ml.libwebsockets.org/pipermail/libwebsockets/2013-
>>> April/000403.html>>
>>>
>>>
>>>
>>>
>>>              . At that time, he was thinking of creating a mutex on
>>>
>>>                          libwebsocket_service. Do you think this is the
>>>         right
>>>                          solution?
>>>
>>>                          Thanks in advance for your help Best regards,
>>>         Thomas
>>>
>>>                      ___________________________________________________
>>>         Libwebsockets
>>>                      mailing list Libwebsockets at ml.__libwebsocke__ts.org
>>>         <http://libwebsockets.org>
>>>                      <mailto:Libwebsockets at ml.__libwebsockets.org
>>>         <mailto:Libwebsockets at ml.libwebsockets.org>>
>>>         http://ml.libwebsockets.org/____mailman/listinfo/libwebsockets
>>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets>
>>>
>>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>
>>>
>>>
>>>                  ___________________________________________________
>>>         Libwebsockets
>>>                  mailing
>>>                  list Libwebsockets at ml.__libwebsocke__ts.org
>>>         <http://libwebsockets.org>
>>>                  <mailto:Libwebsockets at ml.__libwebsockets.org
>>>         <mailto:Libwebsockets at ml.libwebsockets.org>>
>>>         http://ml.libwebsockets.org/____mailman/listinfo/libwebsockets
>>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets>
>>>
>>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>
>>>
>>>
>>>              ___________________________________________________
>>>              Libwebsockets mailing list
>>>              Libwebsockets at ml.__libwebsocke__ts.org
>>>         <http://libwebsockets.org>
>>>              <mailto:Libwebsockets at ml.__libwebsockets.org
>>>         <mailto:Libwebsockets at ml.libwebsockets.org>>
>>>         http://ml.libwebsockets.org/____mailman/listinfo/libwebsockets
>>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets>
>>>
>>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>
>>>
>>>
>>>
>>>
>> _______________________________________________
>> Libwebsockets mailing list
>> Libwebsockets at ml.libwebsockets.org
>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20131223/266b4d76/attachment-0001.html>


More information about the Libwebsockets mailing list