[Libwebsockets] trouble with external POLL array handling

Edwin van den Oetelaar oetelaar.automatisering at gmail.com
Mon Jan 14 19:39:08 CET 2013


I have not solved the problem yet, but already made some changes that
help a little.
I tried to make the handling of the array more efficient :
I just move the last item of the array into the place of the deleted
item and invalidate the last item (so it will surely be ignored by
poll() later)

case LWS_CALLBACK_DEL_POLL_FD:
        fprintf(stderr, "DEL_POLL_FD %d\n", (int) (long) user);
        for (n = 0; n < count_pollfds; n++) {
            if (pollfds[n].fd == (int) (long) user) {
                // found it, copy last item into this space
                // then shorten the array
                if (count_pollfds > 1) {
                    int lastitem = count_pollfds - 1;
                    pollfds[n] = pollfds[lastitem];
                    pollfds[lastitem].fd = -1; // invalidate
                    count_pollfds--;
                } else {
                    // the last item
                    pollfds[n].fd = -1; // invalidate
                    count_pollfds--;
                }
                break; // break from for loop, we have done the work
            }
        }
        break;


This is the result from apache benchmark, using 1000 concurrent
connnections, as you will see, it works very nice, but still shows
starvation on some sockets.
This is because we still skip a socket sometimes when a socket is
deleted in the loop.


$ ab   -t 100 -n 100000 -c 1000 localhost:8088/

Server Software:        libwebsockets
Server Hostname:        localhost
Server Port:            8088

Document Path:          /
Document Length:        0 bytes

Concurrency Level:      1000
Time taken for tests:   6.249 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Non-2xx responses:      100170
Total transferred:      4307310 bytes
HTML transferred:       0 bytes
Requests per second:    16002.36 [#/sec] (mean)
Time per request:       62.491 [ms] (mean)
Time per request:       0.062 [ms] (mean, across all concurrent requests)
Transfer rate:          673.12 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4   36 277.5      9    3011
Processing:     8   14  28.2     13    1768
Waiting:        3   11  28.2     10    1764
Total:         16   50 290.6     22    4777

Percentage of the requests served within a certain time (ms)
  50%     22
  66%     24
  75%     25
  80%     27
  90%     29
  95%     31
  98%     33
  99%     37
 100%   4777 (longest request)


On Mon, Jan 14, 2013 at 4:49 PM, Edwin van den Oetelaar
<oetelaar.automatisering at gmail.com> wrote:
> Summary:
>
> in the handling of "libwebsocket_service_fd(context,  &pollfds[n]))"
> the poll array is changed which lead to trouble, eg. unhandled sockets.
>
> Is anyone else having trouble with this? How to handle it, would using
> epoll() help?
> If you need more info about this, please just ask.
>
> Thanks for your time and effort already,
> Best regards, and I think it is a beautiful piece of code anyways.
> Edwin
>
>
> snip of code from :
> /* libwebsockets-test-server-extpoll - libwebsockets external poll loop sample
> /*
>          * callbacks for managing the external poll() array appear in
>          * protocol 0 callback
>          */
>
>         case LWS_CALLBACK_ADD_POLL_FD:
>                 pollfds[count_pollfds].fd = (int)(long)user;
>                 pollfds[count_pollfds].events = (int)len;
>                 pollfds[count_pollfds++].revents = 0;
>                 break;
>
>         case LWS_CALLBACK_DEL_POLL_FD:
>                 for (n = 0; n < count_pollfds; n++)
>                         if (pollfds[n].fd == (int)(long)user)
>                                 while (n < count_pollfds) {
>                                         pollfds[n] = pollfds[n + 1]; // << TROUBLE
>                                         n++;
>                                 }
>                 count_pollfds--; // << TROUBLE
>                 break;
>
>         case LWS_CALLBACK_SET_MODE_POLL_FD:
>                 for (n = 0; n < count_pollfds; n++)
>                         if (pollfds[n].fd == (int)(long)user)
>                                 pollfds[n].events |= (int)(long)len;
>                 break;
>
>         case LWS_CALLBACK_CLEAR_MODE_POLL_FD:
>                 for (n = 0; n < count_pollfds; n++)
>                         if (pollfds[n].fd == (int)(long)user)
>                                 pollfds[n].events &= ~(int)(long)len;
>                 break;
>
> // later the mainloop is checking
> // but the pollfds[] array changed, leading to fd's not being handled,
> since they moved to a lower index
>
> n = poll(pollfds, count_pollfds, 25);
>                 if (n < 0)
>                         goto done;
>
>                 if (n)
>                         for (n = 0; n < count_pollfds; n++)
>                                 if (pollfds[n].revents)
>                                         /*
>                                         * returns immediately if the fd does not
>                                         * match anything under libwebsockets
>                                         * control
>                                         */
>                                         if (libwebsocket_service_fd(context,
>                                                                   &pollfds[n]))
>                                                 goto done;
>
> // -- end of code



More information about the Libwebsockets mailing list