[Libwebsockets] Setting up a high-throughput libwebsockets server.

Chris Evans chris.evans at adroll.com
Tue Oct 4 04:33:32 CEST 2016


Thanks for the quick reply,

Re allocations: Heap allocations during the initial setup/handshake are
fine so long as there are no allocations per query. (The connection will be
long-running but serve 10,000K queries/sec)

Re echo-server: `libwebsockets-test-echo` seems to behave how I would
expect an "echo server" to. i.e. if I connect to and query it with iocat (
https://www.npmjs.com/package/iocat -- tool like netcat but speaking the
websockets protocol), it echos back my query. (Though I understand if it
wasn't built for performance testing)

I've attached the pcap output from `tcpdump -i lo0 -c 200 -w output.pcap
src port 6666 or dst port 6666` (running the same client/server commands as
before). I'm not too familiar with networking/tcpdump/wireshark but it
looks like the handshake goes down, a few "Test Message"s are echoed, and
then the server resets the connection.

I also added print statements to print the `lws_callback_reason` in the
callback (First 250 lines of output attached as stdout.txt) which seem to
confirm the reset/reconnects from the server's point of view.

Thanks again for the help,
  Chris

On Mon, Oct 3, 2016 at 6:11 PM, Andy Green <andy at warmcat.com> wrote:

> On Mon, 2016-10-03 at 14:22 -0700, Chris Evans wrote:
> > Howdy,
> >
> > I'm looking to set up a simple query/response websockets-protocol
> > server in a high-throughput environment: Serving a machine learning
> > model behind Amazon's new Application Load Balancer (which
> > necessitates speaking the WebSockets protocol) . `libwebsockets`
> > seems a good choice as 1) I'm not looking to build a full web app,
> > and 2) It doesn't allocate per query (is that correct?)
>
> Depends on what you do while processing the query... by default it
> allocates some space at the connection and ws protocol negotiation and
> doesn't need the heap otherwise.
>
> > But I've run into issues when stress testing with the `tcpkali` tool
> > (https://github.com/machinezone/tcpkali), even against the provided
> > test-echo server; `tcpkali` is unable to make a connection (I've had
> > success with tcpkali in other contexts):
>
> The echo server is just designed to talk to himself in client mode.
>  When I wrote it, I didn't realize "echo" has a bigger meaning for
> people looking to make performance measurements.  So it may not do what
> you want for that.
>
> I tried to install tcpkali but it's not packaged in Fedora.
>
> > ```
> > chrisevans at CHRIS-EVANS:bin ((detached from v2.0.0))$ ./libwebsockets-
> > test-echo --port 6666
> > [2016/09/03 13:44:51:3543] NOTICE: Built to support client operations
> > [2016/09/03 13:44:51:3543] NOTICE: Built to support server operations
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: libwebsockets test
> > server echo - license LGPL2.1+SLE
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: (C) Copyright
> > 2010-2016 Andy Green <andy at warmcat.com>
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: Running in server
> > mode
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: Initial logging
> > level 7
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: Libwebsockets
> > version: 2.0.0 chrisevans at CHRIS-EVANS-v2.0.0
>
> You should use v2.0-stable branch or master, there are many fixes since
> v2.0, although I don't think that's the problem.
>
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: IPV6 not compiled
> > in
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: libev support not
> > compiled in
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: libuv support not
> > compiled in
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>:  Threads: 1 each
> > 10240 fds
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>:  mem: platform fd
> > map: 81920 bytes
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>:  Compiled with
> > OpenSSL support
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>:  SSL disabled: no
> > LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>: Creating Vhost
> > 'default' port 6666, 1 protocols
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>:  Listening on port
> > 6666
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>:  mem: per-conn:
> >       488 bytes + protocol rx buf
> > Oct  3 13:44:51 CHRIS-EVANS lwsts[14575] <Notice>:
> > canonical_hostname = CHRIS-EVANS
> > ```
> >
> > ```
> > chrisevans at CHRIS-EVANS:Clients (test_suite)$ tcpkali -m "Test
> > Message" -c 1 --ws  127.0.0.1:6666
> > Destination: [127.0.0.1]:6666
> > Interface lo0 address [127.0.0.1]:0
> > Interface lo0 address [169.254.169.254]:0
> > Using interface lo0 to connect to [127.0.0.1]:6666
> > Could not create 1 connection in allotted time (10s)
> > ```
> > Any help would be greatly appreciated!
>
> This doesn't give me anything to go on... if you can tcpdump what's
> happening on the port, and / or increase the logging (eg -d 65535) at
> the echo server, we might get a clue.
>
> -Andy
>
>
> >
> > -Chris
> > _______________________________________________
> > Libwebsockets mailing list
> > Libwebsockets at ml.libwebsockets.org
> > http://libwebsockets.org/mailman/listinfo/libwebsockets
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20161003/45bd681c/attachment-0001.html>
-------------- next part --------------
[2016/09/03 19:15:28:1638] NOTICE: Initial logging level 7
[2016/09/03 19:15:28:1638] NOTICE: Libwebsockets version: 2.0.0 chrisevans at CHRIS-EVANS-v2.0.0
[2016/09/03 19:15:28:1638] NOTICE: IPV6 not compiled in
[2016/09/03 19:15:28:1638] NOTICE: libev support not compiled in
[2016/09/03 19:15:28:1638] NOTICE: libuv support not compiled in
[2016/09/03 19:15:28:1639] NOTICE:  Threads: 1 each 10240 fds
[2016/09/03 19:15:28:1640] NOTICE:  mem: platform fd map: 81920 bytes
[2016/09/03 19:15:28:1641] NOTICE:  Compiled with OpenSSL support
[2016/09/03 19:15:28:1641] NOTICE:  SSL disabled: no LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT
[2016/09/03 19:15:28:1641] NOTICE: Creating Vhost 'default' port 6666, 1 protocols
[2016/09/03 19:15:28:1652] NOTICE:  Listening on port 6666
[2016/09/03 19:15:28:1652] NOTICE:  mem: per-conn:          488 bytes + protocol rx buf
[2016/09/03 19:15:28:1652] NOTICE:  canonical_hostname = CHRIS-EVANS
Started with options:
Options(6666, 10)
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_ADD_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_PROTOCOL_INIT
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_NETWORK_CONNECTION
LWS_CALLBACK_WSI_CREATE
LWS_CALLBACK_SERVER_NEW_CLIENT_INSTANTIATED
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_ADD_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION
LWS_CALLBACK_ESTABLISHED
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_SERVER_WRITEABLE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_SERVER_WRITEABLE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_DEL_POLL_FD
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CLOSED
LWS_CALLBACK_WSI_DESTROY
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_NETWORK_CONNECTION
LWS_CALLBACK_WSI_CREATE
LWS_CALLBACK_SERVER_NEW_CLIENT_INSTANTIATED
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_ADD_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION
LWS_CALLBACK_ESTABLISHED
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_SERVER_WRITEABLE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_SERVER_WRITEABLE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_DEL_POLL_FD
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CLOSED
LWS_CALLBACK_WSI_DESTROY
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_NETWORK_CONNECTION
LWS_CALLBACK_WSI_CREATE
LWS_CALLBACK_SERVER_NEW_CLIENT_INSTANTIATED
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_ADD_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION
LWS_CALLBACK_ESTABLISHED
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_SERVER_WRITEABLE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_SERVER_WRITEABLE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_DEL_POLL_FD
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CLOSED
LWS_CALLBACK_WSI_DESTROY
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_NETWORK_CONNECTION
LWS_CALLBACK_WSI_CREATE
LWS_CALLBACK_SERVER_NEW_CLIENT_INSTANTIATED
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_ADD_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION
LWS_CALLBACK_ESTABLISHED
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_SERVER_WRITEABLE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_SERVER_WRITEABLE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_DEL_POLL_FD
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CLOSED
LWS_CALLBACK_WSI_DESTROY
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_NETWORK_CONNECTION
LWS_CALLBACK_WSI_CREATE
LWS_CALLBACK_SERVER_NEW_CLIENT_INSTANTIATED
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_ADD_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION
LWS_CALLBACK_ESTABLISHED
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_RECEIVE
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_LOCK_POLL
LWS_CALLBACK_CHANGE_MODE_POLL_FD
LWS_CALLBACK_UNLOCK_POLL
LWS_CALLBACK_GET_THREAD_ID
LWS_CALLBACK_LOCK_POLL
-------------- next part --------------
A non-text attachment was scrubbed...
Name: output.pcap
Type: application/octet-stream
Size: 1569568 bytes
Desc: not available
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20161003/45bd681c/attachment-0001.obj>


More information about the Libwebsockets mailing list