[Libwebsockets] Libwebsocket Upgrade from 1.4

Brice Hamon normandviking at gmail.com
Tue Jul 5 22:28:03 CEST 2016


Thanks Roger,

I'll look into it if I am unable to find the root cause otherwise.

A little bit of progress.

1) The test server and client do work on my platform. So the problem is in
my code somewhere.

2) I changed the client's test program and now I can see that the client
get connected to the server, but nothing after that.
That rules out a network transport problem.


libwebsockets test client
SSL Usage: Yes
[1467750112:5381] NOTICE: Initial logging level 65535
[1467750112:5382] NOTICE: Libwebsockets version: 1.7.0 ac4b079
[1467750112:5382] NOTICE: IPV6 not compiled in
[1467750112:5383] NOTICE: libev support not compiled in
[1467750112:5383] INFO:  LWS_DEF_HEADER_LEN    : 1024
[1467750112:5383] INFO:  LWS_MAX_PROTOCOLS     : 5
[1467750112:5384] INFO:  LWS_MAX_SMP           : 32
[1467750112:5384] INFO:  SPEC_LATEST_SUPPORTED : 13
[1467750112:5384] INFO:  sizeof (*info)        : 216
[1467750112:5385] INFO:  SYSTEM_RANDOM_FILEPATH: '/dev/urandom'
[1467750112:5385] INFO:  default timeout (secs): 20
[1467750112:5386] NOTICE:  Threads: 1 each 1024 fds
[1467750112:5387] INFO:  mem: context:          9800 bytes (5704 ctx + (1
thr x 4096))
[1467750112:5387] INFO:  mem: http hdr rsvd:   67712 bytes (1 thr x (1024 +
3208) x 16))
[1467750112:5388] INFO:  mem: pollfd map:       8192
[1467750112:5388] NOTICE:  mem: platform fd map:  8192 bytes
[1467750112:5389] INFO:  LWS_MAX_EXTENSIONS_ACTIVE: 2
[1467750112:5389] NOTICE:  mem: per-conn:          376 bytes + protocol rx
buf
[1467750112:5390] NOTICE:  canonical_hostname = bs1
callback_bsws [LWS_CALLBACK_OPENSSL_LOAD_EXTRA_CLIENT_VERIFY_CERTS] fd=0
user=[�`ơ#] len=0
callback_bsws [LWS_CALLBACK_PROTOCOL_INIT] fd=0 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_CHANGE_MODE_POLL_FD] fd=-1 user=[NULL] len=0
libbswslog: [LOG_INFO]: lws_client_connect: direct conn

libbswslog: [LOG_INFO]: lws_client_connect_2

libbswslog: [LOG_INFO]: lws_client_connect_2: address 192.168.1.202

callback_bsws [LWS_CALLBACK_LOCK_POLL] fd=6 user=[NULL] len=1
callback_bsws [LWS_CALLBACK_ADD_POLL_FD] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_UNLOCK_POLL] fd=6 user=[NULL] len=1
libbswslog: [LOG_INFO]: nonblocking connect retry

callback_bsws [LWS_CALLBACK_LOCK_POLL] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_CHANGE_MODE_POLL_FD] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_UNLOCK_POLL] fd=6 user=[NULL] len=0
Waiting for connect...
libbswslog: [LOG_INFO]: lws_client_connect_2

libbswslog: [LOG_INFO]: lws_client_connect_2: address 192.168.1.202

libbswslog: [LOG_INFO]: connected







Then when I kill the server:

callback_bsws [LWS_CALLBACK_LOCK_POLL] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_CHANGE_MODE_POLL_FD] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_UNLOCK_POLL] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_CLIENT_APPEND_HANDSHAKE_HEADER] fd=6
user=[NULL] len=3794
callback_bsws [LWS_CALLBACK_LOCK_POLL] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_CHANGE_MODE_POLL_FD] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_UNLOCK_POLL] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_CLIENT_APPEND_HANDSHAKE_HEADER] fd=6
user=[NULL] len=3794
callback_bsws [LWS_CALLBACK_LOCK_POLL] fd=6 user=[NULL] len=1
callback_bsws [LWS_CALLBACK_DEL_POLL_FD] fd=6 user=[NULL] len=0
callback_bsws [LWS_CALLBACK_UNLOCK_POLL] fd=6 user=[NULL] len=1
callback_bsws [LWS_CALLBACK_WSI_DESTROY] fd=6 user=[NULL] len=0


Thanks guys.
Brice.


On Tue, Jul 5, 2016 at 3:47 PM, Roger Light <roger at atchoo.org> wrote:

> Hi Brice,
>
> If you've got a working and a non-working version, then a good option
> is to use "git bisect" to find the problem commit.
>
> git clone https://github.com/warmcat/libwebsockets lws-bisect
> cd lws-bisect
> git bisect start
> git bisect good v1.6.0-chrome48-firefox42 # assuming this one works
> git bisect bad v1.7.0 # assuming this version doesn't
>
> git will then checkout a changeset for you to test - so make your
> build directory, run cmake and carry out your tests. If the test is
> successful, run "git bisect good", else run "git bisect bad". You'll
> have to carry out roughly 6 tests. At the end of the process you'll be
> told which commit in libwebsockets produced the error you are seeing -
> assuming that the problem is with libwebsockets of course.
>
> When you're running your tests, make sure you start from a clean build
> and that your own client code is also recompiled against the test
> version.
>
> Best of luck,
>
> Roger
>
>
>
> On Tue, Jul 5, 2016 at 6:48 PM, Brice Hamon <normandviking at gmail.com>
> wrote:
> > Thanks Andy,
> >
> > I will report my progress.
> >
> > The "Connection refused" is reported by the browser (Chrome), so as you
> > said, it could be something not transport related, but rather SSL as  I
> am
> > inclined to think.
> >
> > Thanks
> > Brice.
> >
> > On Tue, Jul 5, 2016 at 1:44 PM, Andy Green <andy at warmcat.com> wrote:
> >>
> >>
> >>
> >> On July 5, 2016 11:40:30 PM GMT+08:00, Brice Hamon
> >> <normandviking at gmail.com> wrote:
> >> >Hi Andy,
> >> >
> >> >I have not made progress on finding where the problem is.
> >> >
> >> >One thing I was doing and seemed strange was in the SSL certs side:
> >> >
> >> >if (http_ssl_)
> >> >    {
> >> >        if (!chain_file_.empty())
> >> >        {
> >> >          // cert file is not needed as it's included in the chain with
> >> >all intermediate cert.
> >> >            //info.ssl_cert_filepath = cert_file_.c_str();
> >> >            info.ssl_private_key_filepath = key_file_.c_str();
> >> >            info.ssl_cert_filepath = chain_file_.c_str();
> >> >        }
> >> >        else
> >> >        {
> >> >            // Simple key/cert WSS
> >> >            info.ssl_cert_filepath = cert_file_.c_str();
> >> >            info.ssl_private_key_filepath = key_file_.c_str();
> >> >        }
> >> >    }
> >> >    else
> >> >    {
> >> >        info.ssl_cert_filepath = NULL;
> >> >        info.ssl_private_key_filepath = NULL;
> >> >    }
> >> >
> >> >    info.gid = -1;
> >> >    info.uid = -1;
> >> >    info.options = 0;
> >> >    info.ka_time = 10; // keep alive every 10
> >> >    info.ka_probes = 1; // how many times to try to get response before
> >> >giving up
> >> >    info.ka_interval = 2; // how long to wait before each ka_probes
> >> >    info.user = this; // to get ourself inthe static callback via the
> >> >context.
> >> >
> >> >    context_ = libwebsocket_create_context(&info);
> >> >
> >> >
> >> >and this was the only way I made the lib work with 3 ssl files (key,
> >> >cert
> >> >and chain).
> >> >
> >> >I tried the "correct" way with
> >> >
> >> >info.ssl_cert_filepath        = cert_file_.c_str();
> >> >info.ssl_private_key_filepath = key_file_.c_str();
> >> >info.ssl_ca_filepath          = chain_file_.c_str();
> >> >
> >> >With no luck.
> >> >
> >> >Any clues?
> >>
> >> Yeah your biggest clue is "Connection Refused".
> >>
> >> Also you should sanity-check the lws test server itself works on your
> >> toolchain / platform.
> >>
> >> >I think if we find the problem in 1.7, 2.0 will work as well.
> >>
> >> I understand it looks like that to you, since you try 1.6 and it works
> and
> >> 1.7 acts different.  But it's not clear the problem is in the lib.
> It's not
> >> even clear to me what the problem is.
> >>
> >> If it was happening to me I would grab hold of the Connection Refused
> end
> >> of it, eg look what netstat says and eyeball traffic with tcpdump.  The
> lib
> >> doesn't issue a tcp connection refused, as I explained it means
> something
> >> basic like the listen socket is not there.
> >>
> >> So the first move is confirm if that is literally a tcp connection
> >> refused, or you mean some handwaving 'communication failed' error that
> is
> >> actually something totally different and a red herring.
> >>
> >> Build it in debug mode
> >>
> >> cmake -DCMAKE_BUILD_TYPE=DEBUG
> >>
> >> and crank the logs up, equivalent to -d65535 on the test apps and look
> in
> >> there both at init and at connection attempt.  Compare the logs for test
> >> server in the same cases.
> >>
> >> -Andy
> >>
> >> >Thank you in advance,
> >> >Brice.
> >> >
> >> >
> >> >On Mon, Jul 4, 2016 at 5:29 PM, Brice Hamon <normandviking at gmail.com>
> >> >wrote:
> >> >
> >> >> Ok 1.7.0 breaks the server. All version before are OK.
> >> >>
> >> >> The only change I can see impacting my code was the warning:
> >> >>
> >> >> bswebsocket.cpp:155:23: warning: `const lws_extension*
> >> >> lws_get_internal_extensions()' is deprecated (declared at
> >> >> /export/home/development/3rdparty/include/libwebsockets.h:1877)
> >> >> [-Wdeprecated-declarations]
> >> >>
> >> >> So I am investigating in that side.
> >> >>
> >> >> On Mon, Jul 4, 2016 at 4:25 PM, Brice Hamon <normandviking at gmail.com
> >
> >> >> wrote:
> >> >>
> >> >>> Hi Andy,
> >> >>>
> >> >>> I am working backwards in version. So far 2.0.2 to 1.7.2 does not
> >> >work.
> >> >>> 1.5.1 does work. I will eventually where it breaks.
> >> >>>
> >> >>> I'll keep you posted.
> >> >>>
> >> >>> On Thu, Jun 30, 2016 at 1:13 PM, Brice Hamon
> >> ><normandviking at gmail.com>
> >> >>> wrote:
> >> >>>
> >> >>>> Thanks Andy,
> >> >>>>
> >> >>>> I do not think it is a networking or environment issue as the same
> >> >code
> >> >>>> running on the same host with 1.4 works and not 2.0.
> >> >>>>
> >> >>>> I just wanted to make sure I did not forgot an new call to init SSL
> >> >or
> >> >>>> something like that not present in 1.4.
> >> >>>>
> >> >>>> I'll dig into it and report back.
> >> >>>>
> >> >>>> Thanks again
> >> >>>> Brice.
> >> >>>>
> >> >>>> On Wed, Jun 29, 2016 at 1:56 PM, Andy Green <andy at warmcat.com>
> >> >wrote:
> >> >>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On June 29, 2016 11:52:31 PM GMT+08:00, Brice Hamon <
> >> >>>>> normandviking at gmail.com> wrote:
> >> >>>>> >Hi Andy,
> >> >>>>> >
> >> >>>>> >I just upgraded my server from 1.4 to 2.0.
> >> >>>>> >
> >> >>>>> >Not really a big task, just lws_ and a couple of adjustments,
> >> >>>>> >
> >> >>>>> >But now my server does not work anymore.  I use an external
> >> >polling
> >> >>>>> >mechanism, and I see the first FD_ADD so the epoll is working.
> >> >>>>>
> >> >>>>> That doesn't follow actually, it means only that lws' code to
> >> >expose
> >> >>>>> poll fds changes to the callback is working; it's caused by
> >> >creating the
> >> >>>>> listen socket during init, not by servicing (e)poll events.  The
> >> >log
> >> >>>>> doesn't show any evidence of servicing (e)poll events.
> >> >>>>>
> >> >>>>> >But when I try to connect with a browser or client program I get
> >> >a CONN
> >> >>>>> >REFUSED and I don't see anything happening on the server side. I
> >> >just
> >> >>>>> >see
> >> >>>>> >it listening on the correct port.
> >> >>>>>
> >> >>>>> Connection refused is really specific, it means for the peer there
> >> >was
> >> >>>>> nobody listening at the port.  That can mean the listen socket
> >> >bound to a
> >>
> >> This observation is still true if it's literally "Connection Refused".
> >>
> >> -Andy
> >>
> >> >>>>> different interface than your client reached (check it with
> >> >netstat -pltn
> >> >>>>> at the server, 0.0.0.0 means all interfaces) or a firewall got in
> >> >the way,
> >> >>>>> or you reached the wrong ip (name resolution issue at client).
> >> >>>>>
> >> >>>>> >I compiled the lib plain vanilla with no options.
> >> >>>>> >
> >> >>>>> >Here is what I get as startup from your lib:
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >16-06-29 11:38:24.694159 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >Initial
> >> >>>>> >logging
> >> >>>>> >level 15
> >> >>>>> >16-06-29 11:38:24.694176 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >>>>> >Libwebsockets
> >> >>>>> >version: 2.0.0 xxxx at xxxx-v2.0.0-95-ge943a02
> >> >>>>> >16-06-29 11:38:24.694181 DEBUG: [WEBSOCKET ]: libwebsocket: IPV6
> >> >not
> >> >>>>> >compiled in
> >> >>>>> >16-06-29 11:38:24.694193 DEBUG: [WEBSOCKET ]: libwebsocket: libev
> >> >>>>> >support
> >> >>>>> >not compiled in
> >> >>>>> >16-06-29 11:38:24.694198 DEBUG: [WEBSOCKET ]: libwebsocket: libuv
> >> >>>>> >support
> >> >>>>> >not compiled in
> >> >>>>> >16-06-29 11:38:24.694356 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >Threads: 1
> >> >>>>> >each 1024 fds
> >> >>>>> >16-06-29 11:38:24.694453 DEBUG: [WEBSOCKET ]: libwebsocket:  mem:
> >> >>>>> >platform
> >> >>>>> >fd map:  8192 bytes
> >> >>>>> >16-06-29 11:38:24.694526 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >Compiled
> >> >>>>> >with
> >> >>>>> >OpenSSL support
> >> >>>>> >16-06-29 11:38:24.695057 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >Creating
> >> >>>>> >Vhost
> >> >>>>> >'default' port 7682, 2 protocols, IPv6 off
> >> >>>>> >16-06-29 11:38:24.695102 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >Using SSL
> >> >>>>> >mode
> >> >>>>> >16-06-29 11:38:24.698961 DEBUG: [WEBSOCKET ]: libwebsocket:  SSL
> >> >ECDH
> >> >>>>> >curve
> >> >>>>> >'prime256v1'
> >> >>>>> >16-06-29 11:38:24.699011 INFO : [WEBSOCKET ]: callback_http
> >> >>>>> >LWS_CALLBACK_OPENSSL_LOAD_EXTRA_SERVER_VERIFY_CERTS fd=0
> >> >>>>> >context=0x7efff0002800 user=[`uZ&] len=0
> >> >>>>> >16-06-29 11:38:24.699021 ERROR: [WEBSOCKET ]: callback_http: case
> >> >>>>> >LWS_CALLBACK_OPENSSL_LOAD_EXTRA_SERVER_VERIFY_CERTS not handled.
> >> >>>>> >16-06-29 11:38:24.700680 INFO : [WEBSOCKET ]: callback_http
> >> >>>>> >LWS_CALLBACK_LOCK_POLL fd=85 context=0x7efff0002800 user=[NULL]
> >> >len=1
> >> >>>>> >16-06-29 11:38:24.700719 INFO : [WEBSOCKET ]: callback_http
> >> >>>>> >LWS_CALLBACK_ADD_POLL_FD fd=85 context=0x7efff0002800 user=[NULL]
> >> >len=0
> >> >>>>> >16-06-29 11:38:24.700758 INFO : [WEBSOCKET ]: callback_http
> >> >>>>> >LWS_CALLBACK_UNLOCK_POLL fd=85 context=0x7efff0002800 user=[NULL]
> >> >len=1
> >> >>>>> >16-06-29 11:38:24.700811 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >Listening
> >> >>>>> >on
> >> >>>>> >port 7682
> >> >>>>> >16-06-29 11:38:24.700834 DEBUG: [WEBSOCKET ]: libwebsocket:  mem:
> >> >>>>> >per-conn:
> >> >>>>> >         496 bytes + protocol rx buf
> >> >>>>> >16-06-29 11:38:24.700919 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >>>>> > canonical_hostname = xxxx
> >> >>>>> >
> >> >>>>> >Then a
> >> >>>>> >
> >> >>>>> >16-06-29 11:38:25.972014 DEBUG: [WEBSOCKET ]: libwebsocket:
> >> >>>>> >lws_protocol_init
> >> >>>>> >16-06-29 11:38:25.972043 INFO : [WEBSOCKET ]: callback_http
> >> >>>>> >LWS_CALLBACK_PROTOCOL_INIT fd=0 context=0x7efff0002800
> >> >user=[NULL]
> >> >>>>> >len=0
> >> >>>>> >16-06-29 11:38:25.972049 INFO : [WEBSOCKET ]: callback_http:
> >> >>>>> >LWS_CALLBACK_PROTOCOL_INIT
> >> >>>>> >
> >> >>>>> >And nothing else after that.
> >> >>>>>
> >> >>>>> Connection Refused is a tcp level thing, it means you did not
> >> >succeed
> >> >>>>> to touch the listen socket.  So there would be no sign of activity
> >> >from lws
> >> >>>>> about that.
> >> >>>>>
> >> >>>>> >I guess  my straight port from 1.4 to 2.0 was just been
> >> >optimistic. I
> >> >>>>> >am
> >> >>>>> >forgetting something essential in order to have 2.0 working like
> >> >in
> >> >>>>> >1.4.
> >> >>>>>
> >> >>>>> I think either lws bound to the wrong network interface for
> >> >listen, or
> >> >>>>> the issue is outside lws.
> >> >>>>>
> >> >>>>> >Also the only question I had during the port is that I used the
> >> >>>>> >context->user pointer to store my own websocket object. I now
> >> >need to
> >> >>>>> >to a
> >> >>>>> >// Getting the context from the wsi
> >> >>>>> >struct lws_context *wsicont = lws_get_context(wsi);
> >> >>>>> >
> >> >>>>> >// Getting the user cd from the context
> >> >>>>> >BSWebSocket *websocket = (BSWebSocket
> >> >*)lws_context_user(wsicont);
> >> >>>>> >
> >> >>>>> >Is there a better way to do that? I already use the user pointer
> >> >of the
> >> >>>>> >callback to get my per_session_data structure.
> >> >>>>>
> >> >>>>> No it looks good.  Both of those apis are really cheap, just
> >> >resolve to
> >> >>>>> a struct member dereference each.
> >> >>>>>
> >> >>>>> -Andy
> >> >>>>>
> >> >>>>> >Thank you,
> >> >>>>> >
> >> >>>>> >Brice.
> >> >>>>> >
> >> >>>>> >
> >> >>>>> >On Thu, Jun 2, 2016 at 8:48 PM, Brice Hamon
> >> ><normandviking at gmail.com>
> >> >>>>> >wrote:
> >> >>>>> >
> >> >>>>> >> Thank you Andy,
> >> >>>>> >>
> >> >>>>> >> I will get going and report my findings.
> >> >>>>> >>
> >> >>>>> >> Thanks again.
> >> >>>>> >> Brice.
> >> >>>>> >>
> >> >>>>> >>
> >> >>>>> >> On Thu, Jun 2, 2016 at 6:20 PM, Andy Green <andy at warmcat.com>
> >> >wrote:
> >> >>>>> >>
> >> >>>>> >>>
> >> >>>>> >>>
> >> >>>>> >>> On June 3, 2016 1:08:32 AM GMT+08:00, Brice Hamon <
> >> >>>>> >>> normandviking at gmail.com> wrote:
> >> >>>>> >>> >Hi all,
> >> >>>>> >>> >
> >> >>>>> >>> >I have been using libwebsocket 1.4 with external epoll
> >> >notification
> >> >>>>> >>> >loop
> >> >>>>> >>> >without a single problem for over a year now.
> >> >>>>> >>>
> >> >>>>> >>> Great.
> >> >>>>> >>>
> >> >>>>> >>> >All the libwebsocket functions are happening in one thread so
> >> >I
> >> >>>>> >have
> >> >>>>> >>> >followed all guidelines in the README.coding
> >> >>>>> >>> >from that version.
> >> >>>>> >>> >
> >> >>>>> >>> >I think it is now time for me to upgrade as, following this
> >> >group,
> >> >>>>> >many
> >> >>>>> >>> >fix
> >> >>>>> >>> >and enhancement took place.
> >> >>>>> >>> >
> >> >>>>> >>> >Is there specific areas of change I should be concerned with,
> >> >or
> >> >>>>> >>> >important
> >> >>>>> >>> >changes in the notification area?
> >> >>>>> >>>
> >> >>>>> >>> Yes at 1.6 many api names got simplified and rationalized.
> >> >Mainly
> >> >>>>> >the
> >> >>>>> >>> api prefix got standardized to 'lws_'.  There are 4 steps
> >> >listed in
> >> >>>>> >the
> >> >>>>> >>> changelog
> >> >>>>> >>>
> >> >>>>> >>>
> >> >>>>> >
> >> >>>>>
> >> >
> https://github.com/warmcat/libwebsockets/blob/v2.0-stable/changelog#L643
> >> >>>>> >>>
> >> >>>>> >>> to search / replace everything in your code into shape.
> >> >>>>> >>>
> >> >>>>> >>> There are many other changes and improvements, such as
> >> >multiple
> >> >>>>> >vhost
> >> >>>>> >>> support, CGI support, protocol plugins, lwsws etc.  It's hard
> >> >to
> >> >>>>> >predict
> >> >>>>> >>> which are interesting, if you have external epoll() just to
> >> >get
> >> >>>>> >epoll, the
> >> >>>>> >>> new built-in libuv support might be interesting... if so then
> >> >lwsws
> >> >>>>> >and
> >> >>>>> >>> plugins might also simplify your life.
> >> >>>>> >>>
> >> >>>>> >>> If you need external event loop to interface to something
> >> >else, then
> >> >>>>> >that
> >> >>>>> >>> should still work the same.
> >> >>>>> >>>
> >> >>>>> >>> >Thanks all and especially Andy, your lib rocks :)
> >> >>>>> >>>
> >> >>>>> >>> Thanks for the kind words ^^
> >> >>>>> >>>
> >> >>>>> >>> -Andy
> >> >>>>> >>>
> >> >>>>> >>> >Brice.
> >> >>>>> >>> >
> >> >>>>> >>> >
> >> >>>>> >>>
> >> >>>>>
> >> >>>>>
> >>
> >> >>>
> >>>------------------------------------------------------------------------
> >> >>>>> >>> >
> >> >>>>> >>> >_______________________________________________
> >> >>>>> >>> >Libwebsockets mailing list
> >> >>>>> >>> >Libwebsockets at ml.libwebsockets.org
> >> >>>>> >>> >http://libwebsockets.org/mailman/listinfo/libwebsockets
> >> >>>>> >>>
> >> >>>>> >>>
> >> >>>>> >>
> >> >>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >
> >
> > _______________________________________________
> > Libwebsockets mailing list
> > Libwebsockets at ml.libwebsockets.org
> > http://libwebsockets.org/mailman/listinfo/libwebsockets
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20160705/9b9b3993/attachment-0001.html>


More information about the Libwebsockets mailing list