[Libwebsockets] Interaction of external threads with libwebsockets server

"Andy Green (林安廸)" andy at warmcat.com
Sat Dec 21 04:34:28 CET 2013


On 21/12/13 10:49, the mail apparently from "Andy Green (林安廸)" included:
> On 20/12/13 19:38, the mail apparently from Thomas Spitz included:
>> Hello Andy,
>>
>> In fact, in my application, I mostly often only have 0 to 5 active
>> connections at the same time with a maximum number of simultaneous
>> connections restricted to 32 (Besides, why is there no simple solution
>> to reduce the maximum simultaneous connection? I have seen that we need
>> to reduce the max system fds...).
>
> You should be able to do
>
> ulimit -n 64
>
> before starting the server.  It only affects that shell and children. If
> you're bothered by the memory footprint of the parent shell, it should
> be workable to do an exec instead of running as a child.
>
> The default is 1024 on most systems... the way we deal with fd lookups
> it's just not that expensive to allow 1024 (a few KB).
>
>> Therefore, I think I should implement your following proposal using
>> External_Poll I guess:
>>
>>     Poll can only wait on file or socket descriptors... you could
>>     reserve a socket to signal to break out and reload the poll()
>>     because somebody's attribute changed.
>
> Actually I was imagining adding it in lws.

I just pushed a patch

http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=91f19d8d799fdce0459c80357b39930b3ac55a9c

which should give you a single place to signal all pollfd changes from 
if you want to have a go at a patch to implement this on lws side.

If you make a new socket or other descriptor-based thing in the context 
create code that is open for the lifetime of the client or server, and 
add it to the pollfs there with POLLIN event set, writing a byte to it 
from lws_change_pollfd() the first time something changed before the 
outer loop reloads should do what you want.

-Andy

>> I tried to do it but my thread is still not able to break out the
>> poll(). In my test below (using test-server.c as skeletton),  Although I
>
> I think it must mean your test is broken.  Otherwise you're saying that
> poll() doesn't work.
>
> -Andy
>
>> write in the tube that is monitored by poll I see that my asyncSending
>> is never able to break up poll(). poll() always break up on 10s timeout
>> whereas my thread write into the tube every second.
>>
>> int tube[2];
>>
>> (...)
>>
>> void *asyncSending(void *arg) {
>>
>> unsigned char c=0;
>> printf("START asynchronous sending\n");
>>
>> while(1){
>> c++;
>> printf("write in tube\n");
>> if(write(tube[1],&c,1)!=1){
>> printf("Erreur write %s\n", strerror(errno));
>> }
>> //libwebsocket_callback_on_writable_all_protocol(&protocols[PROTOCOL_DUMB_INCREMENT]);
>>
>> }
>>
>> printf("END asynchronous sending\n");
>> return NULL ;
>>
>> }
>>
>>
>>   int main(int argc, char **argv) {
>>
>>          (...)
>>
>> n = 0;
>>
>> printf("Tube creation\n");
>> if(pipe(tube)!=0){
>> printf("Pipe creation error %s\n", strerror(errno));
>> }
>>
>> if (count_pollfds >= max_poll_elements) {
>> lwsl_err("LWS_CALLBACK_ADD_POLL_FD: too many sockets to track\n");
>> return 1;
>> }
>>
>> fd_lookup[tube[0]] = count_pollfds;
>> pollfds[count_pollfds].fd = fd_lookup[tube[0]];
>> pollfds[count_pollfds].events = POLLIN | POLLPRI |POLL_OUT;
>> pollfds[count_pollfds++].revents = 0;
>>
>> int statut_Thread;
>> pthread_t Thread;
>>
>> statut_Thread = pthread_create(&Thread, NULL, asyncSending, context);
>> if (statut_Thread != 0) {
>> printf("Thread creation errror \n");
>> }
>>
>> while (n >= 0 && !force_exit) {
>>
>> /*
>> * this represents an existing server's single poll action
>> * which also includes libwebsocket sockets
>> */
>>
>> n = poll(pollfds, count_pollfds, 10000);//10s
>> if (n < 0)
>> continue;
>>
>> if (n)
>> for (n = 0; n < count_pollfds; n++)
>> if (pollfds[n].revents)
>> /*
>> * returns immediately if the fd does not
>> * match anything under libwebsockets
>> * control
>> */
>> if (libwebsocket_service_fd(context,
>> &pollfds[n]) < 0)
>> goto done;
>>
>>                 (...)
>>         }
>> }
>>
>> Thanks again in advance for your help.
>>
>> BR, Thomas
>>
>>
>> 2013/12/19 "Andy Green (林安廸)" <andy at warmcat.com
>> <mailto:andy at warmcat.com>>
>>
>>     On 19/12/13 02:12, the mail apparently from Thomas Spitz included:
>>
>>         Hello Andy, hello Eugene,
>>
>>         If I understood it correctly, thanks to your patch, we can now
>>         prevent
>>         our external thread(s) from calling
>>         libwebsocket_callback_on___writable
>>         while lws is actively dealing with POLLIN and POLLOUT of fds
>>         using our
>>         own mutex? Thus we are sure that the request(s) from our external
>>         thread(s) will not be lost, or at least trigger
>>         a LWS_CALLBACK_SERVER_WRITEABLE?
>>
>>
>>     I'm not sure it won't get lost, but it won't leave things in an
>>     indeterminate state.  It can still happen the non-service thread
>>     asks to set POLLOUT just as we're about to take the mutex and
>> clear it.
>>
>>     At the moment that'll still break at low probability but it should
>>     be solveable.
>>
>>
>>         My second question is related to performance:
>>
>>         In the documentation it is said
>>
>>              If you want to send something, do not just send it but
>>         request a
>>              callback |when the socket is writeable |
>>
>>         That is why our external thread call
>>         libwebsocket_callback_on___writable
>>         (). Then it is written
>>
>>              |Usually you will get called back immediately next time
>>         around the
>>              service loop|
>>
>>         which seems to mean that  (according to my tests) we have to
>>         wait libwebsocket_service() to end up before the socket is
>> writeable
>>         again. That implies to set a very short timeout
>>
>>
>>     Well... you are right if we have one connection.  But if you have 50
>>     or 50,000 connections, somebody almost always has something going on
>>     before the timeout.
>>
>>     Also in your user callback, you can take the approach like in the
>>     test apps to loop using lws_send_pipe_choked() test to absolutely
>>     stuff your pipe each time you get service.  That doesn't delay
>>     anything looping there because it only loops while the send won't
>>     block.  And it means the kernel is passing out what you stuffed into
>>     the pipe while you wait through any latency in the next "writeable"
>>     service.
>>
>>
>>         for libwebsocket_service(context, VERYSHORTTIMEOUT) otherwise it
>>         will
>>         take a long time before the external thread request could be
>>         handled...
>>         This brings me to wonder why it is not possible to loop on
>>         libwebsocket_service(context, VERYLONGTIMEOUT) that would be
>>         interrupted
>>         by our external thread instead? Thus lws thread is asleeped most
>>         of the
>>         time when there is no incoming IP traffic but can be awaken as
>>         soon as
>>         some outgoing traffic needs to be sent (generated from the
>>         external thread).
>>
>>
>>     Well lws is a singlethreaded nonblocking implementation.  It is
>>     designed to run on small resource-costrained platforms at the same
>>     time a being able to service huge numbers of connections in a
>>     lightweight way.  It does not require threads.
>>
>>     Poll can only wait on file or socket descriptors... you could
>>     reserve a socket to signal to break out and reload the poll()
>>     because somebody's attribute changed.  But in most real usage
>>     scenarios, some other connection is always wanting service before
>>     the timeout anyway and as more connections are seen the less it
>>     matters about latency from waiting for poll() reload.
>>
>>     -Andy
>>
>>         I hope my two questions are clear..
>>
>>         Thanks in advance for your rely.
>>
>>         Best regards, Thomas
>>
>>         2013/12/18 "Andy Green (林安廸)" <andy at warmcat.com
>>         <mailto:andy at warmcat.com>
>>         <mailto:andy at warmcat.com <mailto:andy at warmcat.com>>>
>>
>>
>>              On 18/12/13 06:40, the mail apparently from Andy Green
>>         included:
>>
>>
>>
>>                  Eugene Agafonov <e.a.agafonov at gmail.com
>>         <mailto:e.a.agafonov at gmail.com>
>>                  <mailto:e.a.agafonov at gmail.com
>>         <mailto:e.a.agafonov at gmail.com>__>> wrote:
>>
>>                      Hi!
>>
>>                      I've just implemented the similar scenario just
>>         like it is
>>                      stated
>>                      in document
>>
>> http://git.libwebsockets.org/____cgi-bin/cgit/libwebsockets/____tree/README.coding
>>
>>
>> <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__tree/README.coding>
>>
>>
>>
>>
>> <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__tree/README.coding
>>
>>
>> <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/tree/README.coding>>
>>
>>
>>
>>
>>              I have a mutex-protected queue of outgoing events (as a
>>         std::list of
>>
>>                      strings). Each string is intended to be send in
>>         separated
>>                      message.
>>
>>                      The writer thread posts data onto queue and calls
>>                      libwebsocket_callback_on_____writable(context, wsi)
>>
>>
>>                      Protocol callback gets
>>         LWS_CALLBACK_SERVER_WRITEABLE as soon as
>>                      websocket thread is ready to send data. Once it
>>         comes, the
>>                      callback
>>                      extracts strings from the queue one by one and
>>         writes them to
>>                      socket using libwebsocket_write.
>>
>>                      I'm quite newby in writing WS server with
>>         libwebsockets. Any
>>                      comments from WS gurus are welcome
>>
>>
>>                  What you're doing is a good way to interact with what
>>         lws needs and
>>                  get what you want from threading.
>>
>>                  There's one danger with it, but that is a very tightly
>>         defined race
>>                  rather than corruption or whatever.
>>
>>
>>              Well... there's another thing to take care about... you
>> need to
>>              manage your private list of live wsis carefully.  It means
>>         managing
>>              that list (with any of your locking needed) at the
>>         ESTABLISHED /
>>              CLOSED callback for the WSI.  That should be enough I think.
>>
>>
>>                  The issue is that the 'wait for writeable' is oneshot,
>>         so after it
>>                  becomes true in the poll lws will clear the POLLOUT
>>         wait flag.  But
>>                  because the threads are asynchronous, eventually the
>>         other thread
>>                  will come at the wrong time and set it just as we're
>>         clearing it
>>                  with
>>                  indeterminate results.  Because it's read-modify-write
>>         and flow
>>                  control code also modifies this, there's a low
>>         probability deadlock
>>                  case where POLLIN or POLLOUT from one thread or another
>>         did not get
>>                  set for wait as the thread thinks it is.
>>
>>                  If the extpoll stuff is used the issue can be worse
>>         depending on
>>                  what
>>                  it is doing.
>>
>>                  However the approach you're taking is otherwise really
>>         workable I
>>                  think.  To make it rock solid you'll need to patch
>> lws to
>>                  mutex-protect the few places it changes poll wait flags.
>>
>>
>>              I added a patch
>>
>>
>> http://git.libwebsockets.org/____cgi-bin/cgit/libwebsockets/____commit/?id=____7a1327977ac10bdbace0012274b8ae____889219880e
>>
>>
>> <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e>
>>
>>
>>
>>
>> <http://git.libwebsockets.org/__cgi-bin/cgit/libwebsockets/__commit/?id=__7a1327977ac10bdbace0012274b8ae__889219880e
>>
>>
>> <http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=7a1327977ac10bdbace0012274b8ae889219880e>>
>>
>>
>>              that lets you do this in a lock and unlock callback in
>> the user
>>              code, in one place.
>>
>>              If you're not using threading you don't need to do anything
>>         new.
>>
>>              -Andy
>>
>>
>>                  -Andy
>>
>>                      BR, Eugene Agafonov.
>>
>>                      On Tuesday 17 December 2013 12:03:43 Thomas Spitz
>>         wrote:
>>
>>                          Hello everyone,
>>
>>                          I am trying to find a way for external threads
>>         to ask
>>                          libwebsocket
>>
>>                      server
>>
>>                          to send websocket message on their behalf (as
>> they
>>                          cannot do it
>>
>>                      directly).
>>
>>                          The only way I see at the present is to make a
>>         loop with
>>                          libwebsocket_service (context, VERYSHORTTIME);
>>         and test
>>                          potential
>>
>>                      incoming
>>
>>                          messages from other threads. This is not very
>>         elegant as CPU
>>                          computes
>>
>>                      all
>>
>>                          the time and not very reliable as I should put
>>                          VERYSHORTTIME at 0
>>                          if
>>
>>                      I
>>
>>                          don't want to miss any message.
>>
>>                          I tried the following modification in
>>         test-server.c :
>>
>>                          void *connexionServeur(void *arg) {
>>
>>                              struct libwebsocket_context *context = arg;
>>         while (1) {
>>                              if(libwebsocket_service(____context,
>> 60000)<0){
>>
>>
>>                              printf("Problem!\n");
>>
>>                              break; } }
>>
>>                              return NULL ;
>>
>>                              } int statut_Thread; pthread_t Thread;
>>         statut_Thread =
>>                              pthread_create(&Thread, NULL,
>> connexionServeur,
>>
>>                      context
>>
>>                              );
>>
>>                              if (statut_Thread != 0) {
>>
>>                              printf("Problem with thread creation \n");
>>         } while
>>                              (n >= 0 &&
>>                              !force_exit) { struct timeval tv;
>>         gettimeofday(&tv,
>>                              NULL); if
>>                              (((unsigned int)tv.tv_usec - oldus) >
>> 50000) {
>>
>>
>>
>>
>> libwebsocket_callback_on_____writable_all_protocol(&____protocols[PROTOCOL_DUMB_I
>>
>>
>>
>>
>>
>>              NCREMENT]); oldus = tv.tv_usec;
>>
>>                              libwebsocket_service(context, 0);
>>
>>                              }
>>
>>                              usleep(10000); }
>>
>>
>>                          but it crashes quickly as soon as I try
>> drawing . I
>>                          suppose that
>>                          libwebsocket_service is called twice at the
>>         same time and it
>>                          creates conflicts that lead to stop crash the
>>         websocket.
>>
>>                          I have seen in the mailing list that Thomas
>>         Koeper has
>>                          already
>>                          had a similar question
>>
>>
>> http://ml.libwebsockets.org/____pipermail/libwebsockets/2013-____April/000403.html
>>
>>
>> <http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-__April/000403.html>
>>
>>
>>
>>
>> <http://ml.libwebsockets.org/__pipermail/libwebsockets/2013-__April/000403.html
>>
>>
>> <http://ml.libwebsockets.org/pipermail/libwebsockets/2013-April/000403.html>>
>>
>>
>>
>>
>>              . At that time, he was thinking of creating a mutex on
>>
>>                          libwebsocket_service. Do you think this is the
>>         right
>>                          solution?
>>
>>                          Thanks in advance for your help Best regards,
>>         Thomas
>>
>>                      ___________________________________________________
>>         Libwebsockets
>>                      mailing list Libwebsockets at ml.__libwebsocke__ts.org
>>         <http://libwebsockets.org>
>>                      <mailto:Libwebsockets at ml.__libwebsockets.org
>>         <mailto:Libwebsockets at ml.libwebsockets.org>>
>>         http://ml.libwebsockets.org/____mailman/listinfo/libwebsockets
>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets>
>>
>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>
>>
>>
>>                  ___________________________________________________
>>         Libwebsockets
>>                  mailing
>>                  list Libwebsockets at ml.__libwebsocke__ts.org
>>         <http://libwebsockets.org>
>>                  <mailto:Libwebsockets at ml.__libwebsockets.org
>>         <mailto:Libwebsockets at ml.libwebsockets.org>>
>>         http://ml.libwebsockets.org/____mailman/listinfo/libwebsockets
>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets>
>>
>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>
>>
>>
>>              ___________________________________________________
>>              Libwebsockets mailing list
>>              Libwebsockets at ml.__libwebsocke__ts.org
>>         <http://libwebsockets.org>
>>              <mailto:Libwebsockets at ml.__libwebsockets.org
>>         <mailto:Libwebsockets at ml.libwebsockets.org>>
>>         http://ml.libwebsockets.org/____mailman/listinfo/libwebsockets
>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets>
>>
>>         <http://ml.libwebsockets.org/__mailman/listinfo/libwebsockets
>>         <http://ml.libwebsockets.org/mailman/listinfo/libwebsockets>__>
>>
>>
>>
>
> _______________________________________________
> Libwebsockets mailing list
> Libwebsockets at ml.libwebsockets.org
> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets




More information about the Libwebsockets mailing list