[Libwebsockets] Atomic Frame Size

Duy Lan Nguyen ndlan2k at gmail.com
Sun Apr 12 00:01:13 CEST 2015


I also see libwebsocket_close_and_free_session is called
inside libwebsocket_service_timeout_check. I don't know if that call will
go to LWS_CALLBACK_CLOSED or it will do another timeout above as there is
still something to send?

Many thanks,
Lan

On Sat, Apr 11, 2015 at 2:36 PM, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:

> Playing with lws some more, I think I see why LWS_CALLBACK_CLOSED is
> never reached in that case. In libwebsocket_close_and_free_session, I see
> if (wsi->truncated_send_len) { lwsl_info("wsi %p entering
> WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE\n", wsi); wsi->state =
> WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE; libwebsocket_set_timeout(wsi,
> PENDING_FLUSH_STORED_SEND_BEFORE_CLOSE, 5); return; }
> that means if there is still something to send, it will try to flush it
> rather continuing in libwebsocket_close_and_free_session and eventually
> calling LWS_CALLBACK_CLOSED.
>
> Is it possible to do that code block later with more checking to make sure
> that flushing is worth trying. otherwise just continuing to LWS_CALLBACK_CLOSED?
> Or if you have other suggestions?
>
> Many thanks,
> Lan
>
> On Sat, Apr 11, 2015 at 10:53 AM, Duy Lan Nguyen <ndlan2k at gmail.com>
> wrote:
>
>> Hi Andy,
>>
>> We test how lws client handles server's sudden shutdown and observe the
>> following.
>>
>> We use the method in call_back you advised before to keep the kernel
>> stuffed when sending (do while loop with lws_partial_buffered and
>> lws_send_pipe_choked).
>>
>>     case LWS_CALLBACK_CLIENT_WRITEABLE:
>> do {
>> nTemp1 = (*ppSocketInfo)->m_SendInfo.m_cbSendData -
>> (*ppSocketInfo)->m_SendInfo.m_nBytesSent;
>>
>> nTemp2 = (nTemp1 > ATOM_DATA_SIZE) ? ATOM_DATA_SIZE : nTemp1;
>>
>> if
>> (memcpy_s(&(*ppSocketInfo)->m_SendInfo.m_rgbSendBuffer[LWS_SEND_BUFFER_PRE_PADDING],
>> ATOM_DATA_SIZE,
>> (*ppSocketInfo)->m_SendInfo.m_pbSendData +
>> (*ppSocketInfo)->m_SendInfo.m_nBytesSent,
>> nTemp2))
>> {
>> (*ppSocketInfo)->m_SendInfo.m_retSend = FAIL;
>> goto exit_send;
>> }
>>
>> n = libwebsocket_write(wsi,
>> &(*ppSocketInfo)->m_SendInfo.m_rgbSendBuffer[LWS_SEND_BUFFER_PRE_PADDING],
>> nTemp2,
>> nOpts | LWS_WRITE_BINARY);
>>
>> if (n < 0)
>> {
>> (*ppSocketInfo)->m_SendInfo.m_retSend = FAIL;
>> goto exit_send;
>> }
>>  (*ppSocketInfo)->m_SendInfo.m_nBytesSent += n;
>>
>> if ((*ppSocketInfo)->m_SendInfo.m_nBytesSent >=
>> (*ppSocketInfo)->m_SendInfo.m_cbSendData)
>> {
>> goto exit_send;
>> }
>>
>> }
>>
>> if (lws_partial_buffered(wsi))
>> break;
>> } while (!lws_send_pipe_choked(wsi));
>>
>> In case sending data is smaller (such as 100 KB), I guess this loop
>> completes faster. So when the server shuts down suddenly, the client side
>> immediately goes into case LWS_CALLBACK_CLOSED and stops right away.
>>
>> In case sending data is big (such as 10 MB), the kernel seems never get
>> stuffed (the machine is pretty powerful), and the whole 10 MB just uses the
>> loop of a single case LWS_CALLBACK_CLIENT_WRITEABLE and it takes a bit
>> longer. So when the server shuts down suddenly, although the loop also
>> stops right away, the case LWS_CALLBACK_CLOSED is never reached. And we
>> have to use an external timeout to decide that the sending has failed.
>>
>> Is there anyway that we can have sending bigger data (10MB) behaves the
>> same as sending smaller data when the server suddenly shuts down: going to LWS_CALLBACK_CLOSED
>> right away??
>>
>> Many thanks,
>> Lan
>>
>>
>>
>> On Wed, Apr 1, 2015 at 8:50 AM, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:
>>
>>> Turn out that just removing #include stdint.h is enough and lws still
>>> works fine.
>>>
>>> I made another change to allow using cert blobs instead of using cert
>>> files, with cyassl. I can provide that as well?
>>>
>>> Should I email changes to you or push it to git?
>>>
>>> Thanks
>>>
>>> On Sat, Mar 28, 2015 at 5:34 AM, Andy Green <andy at warmcat.com> wrote:
>>>
>>>>
>>>>
>>>> On 28 March 2015 19:09:53 GMT+08:00, Duy Lan Nguyen <ndlan2k at gmail.com>
>>>> wrote:
>>>> >Hi Andy,
>>>> >
>>>> >Would it be possible to make this change?
>>>>
>>>> Sure, send a patch.
>>>>
>>>> The only windows this gets built on, an autobuilder, it builds already.
>>>>
>>>> If your patch makes it build for you and doesn't break the autobuilder
>>>> then fine...
>>>>
>>>> -Andy
>>>>
>>>> >We'd greatly appreciate!!!
>>>> >
>>>> >On Thu, Mar 26, 2015 at 10:17 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>>>> >wrote:
>>>> >
>>>> >> We need to use Visual Studio 2008 (VS 9) with libwebsockets to be
>>>> >> compatible with our Driver DK. It was ok with previous lws but the
>>>> >latest
>>>> >> lws doesn't compile as libwebsockets.h includes stdint.h that's not
>>>> >yet
>>>> >> available in VS 9. Could you please fix that so that lws will work
>>>> >with VS
>>>> >> 9?
>>>> >>
>>>> >> Thanks!!
>>>> >>
>>>> >> On Wed, Mar 25, 2015 at 1:43 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>>>> >wrote:
>>>> >>
>>>> >>> Awesome. Thanks
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Tue, Mar 24, 2015 at 10:55 AM, Andy Green <andy at warmcat.com>
>>>> >wrote:
>>>> >>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On 23 March 2015 22:28:45 GMT+08:00, Duy Lan Nguyen
>>>> ><ndlan2k at gmail.com>
>>>> >>>> wrote:
>>>> >>>> >Yes, it's SSL connection
>>>> >>>>
>>>> >>>> I just pushed something that should solve it.
>>>> >>>>
>>>> >>>> -Andy
>>>> >>>>
>>>> >>>> >-----Original Message-----
>>>> >>>> >From: "Andy Green" <andy at warmcat.com>
>>>> >>>> >Sent: ‎3/‎23/‎2015 4:02 AM
>>>> >>>> >To: "Duy Lan Nguyen" <ndlan2k at gmail.com>
>>>> >>>> >Cc: "libwebsockets at ml.libwebsockets.org"
>>>> >>>> ><libwebsockets at ml.libwebsockets.org>
>>>> >>>> >Subject: Re: Atomic Frame Size
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >On 23 March 2015 15:57:30 GMT+08:00, Duy Lan Nguyen
>>>> ><ndlan2k at gmail.com>
>>>> >>>> >wrote:
>>>> >>>> >>Hi Andy,
>>>> >>>> >>
>>>> >>>> >>I still have the problem of libwebsocket_context_destroy()
>>>> >hanging,
>>>> >>>> >but
>>>> >>>> >>I
>>>> >>>> >>still don't understand its cause. It happens as follows in my
>>>> >Windows
>>>> >>>> >>machine.
>>>> >>>> >>
>>>> >>>> >>In a client-server connection, the server is suddenly shut down.
>>>> >The
>>>> >>>> >>client
>>>> >>>> >>doesn't realize it after a while, and finally closes down itself.
>>>> >When
>>>> >>>> >>the
>>>> >>>> >>client calls libwebsocket_context_destroy(), it hangs in the
>>>> >following
>>>> >>>> >>loop
>>>> >>>> >>inside libwebsocket_context_destroy()  as wsi is never NULL
>>>> >>>> >>
>>>> >>>> >>for (n = 0; n < context->fds_count; n++) {
>>>> >>>> >>struct libwebsocket *wsi =
>>>> >>>> >>context->lws_lookup[context->fds[n].fd];
>>>> >>>> >>if (!wsi)
>>>> >>>> >>continue;
>>>> >>>> >>libwebsocket_close_and_free_session(context,
>>>> >>>> >>wsi, LWS_CLOSE_STATUS_NOSTATUS /* no protocol close */);
>>>> >>>> >>n--;
>>>> >>>> >>}
>>>> >>>> >>
>>>> >>>> >>This happens as inside libwebsocket_close_and_free_session(), the
>>>> >code
>>>> >>>> >>keeps going into
>>>> >>>> >>
>>>> >>>> >>case WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE:
>>>> >>>> >>if (wsi->truncated_send_len) {
>>>> >>>> >>libwebsocket_callback_on_writable(context, wsi);
>>>> >>>> >>return;
>>>> >>>> >>}
>>>> >>>> >>
>>>> >>>> >>Did I do something wrong here or could you describe a solution to
>>>> >this
>>>> >>>> >
>>>> >>>> >Probably nothing more wrong than using windows.
>>>> >>>> >
>>>> >>>> >>hanging problem?
>>>> >>>> >
>>>> >>>> >I guess there's a case where he has something he wants to send
>>>> >before
>>>> >>>> >dying, but he can't send it.  Is it an SSL connection?
>>>> >>>> >
>>>> >>>> >-Andy
>>>> >>>> >
>>>> >>>> >>Thanks,
>>>> >>>> >>Lan
>>>> >>>> >>
>>>> >>>> >>
>>>> >>>> >>On Sun, Jan 25, 2015 at 8:13 PM, Andy Green <andy at warmcat.com>
>>>> >wrote:
>>>> >>>> >>
>>>> >>>> >>>
>>>> >>>> >>>
>>>> >>>> >>> On 11 January 2015 17:15:56 GMT+08:00, Duy Lan Nguyen
>>>> >>>> >><ndlan2k at gmail.com>
>>>> >>>> >>> wrote:
>>>> >>>> >>> >Hi Andy,
>>>> >>>> >>> >
>>>> >>>> >>> >I've implemented item 3). I can contribute it to lws if you
>>>> >want.
>>>> >>>> >>> >
>>>> >>>> >>> >Do you have any news on items 1) and 2)?
>>>> >>>> >>> >
>>>> >>>> >>> >Thanks,
>>>> >>>> >>> >Lan
>>>> >>>> >>> >
>>>> >>>> >>> >
>>>> >>>> >>> >On Tue, Dec 23, 2014 at 7:26 PM, Duy Lan Nguyen
>>>> ><ndlan2k at gmail.com>
>>>> >>>> >>> >wrote:
>>>> >>>> >>> >
>>>> >>>> >>> >> In 1) and 2), I mean the server goes down suddenly
>>>> >>>> >>> >>
>>>> >>>> >>> >> On Tue, Dec 23, 2014 at 7:17 PM, Duy Lan Nguyen
>>>> >>>> >><ndlan2k at gmail.com>
>>>> >>>> >>> >wrote:
>>>> >>>> >>> >>
>>>> >>>> >>> >>> Hi Andy,
>>>> >>>> >>> >>>
>>>> >>>> >>> >>> I have some questions if you could help
>>>> >>>> >>> >>> 1) I'm trying to let client handle gracefully when the
>>>> >server
>>>> >>>> >>shuts
>>>> >>>> >>> >down.
>>>> >>>> >>> >>> With lots of traffic going back and forth (each of client
>>>> >and
>>>> >>>> >>server
>>>> >>>> >>> >has 2
>>>> >>>> >>> >>> threads, one for sending and one for receiving),
>>>> >>>> >>LWS_CALLBACK_CLOSED
>>>> >>>> >>> >>> sometimes isn't caught by polling, and its callback routine
>>>> >(in
>>>> >>>> >>the
>>>> >>>> >>> >>> switch-case) isn't called. Is it possible to make
>>>> >>>> >>> >LWS_CALLBACK_CLOSED
>>>> >>>> >>> >>> never be dropped?
>>>> >>>> >>>
>>>> >>>> >>> LWS never drops any close message AFAIK, but the OS may never
>>>> >become
>>>> >>>> >>aware
>>>> >>>> >>> that the connection is dead.  It doesn't know if the reason
>>>> >>>> >>everything is
>>>> >>>> >>> quiet is because everything is quiet, or because somewhere a
>>>> >network
>>>> >>>> >>cable
>>>> >>>> >>> fell out.
>>>> >>>> >>>
>>>> >>>> >>> You need to use keepalives of some sort (tcp, or PING / PONG in
>>>> >lws)
>>>> >>>> >>to
>>>> >>>> >>> detect that.
>>>> >>>> >>>
>>>> >>>> >>> >>> 2) And after the connection is already shutdown for a while
>>>> >>>> >>(client
>>>> >>>> >>> >>> doesn't know it as LWS_CALLBACK_CLOSED is dropped), calling
>>>> >>>> >>> >>> libwebsocket_context_destroy() is hanged. Is it possible to
>>>> >>>> >avoid
>>>> >>>> >>it
>>>> >>>> >>> >>> hanging?
>>>> >>>> >>>
>>>> >>>> >>> Sorry where does it hang in there?
>>>> >>>> >>>
>>>> >>>> >>> >>> 3) We want to build a single-file client application, so
>>>> >the app
>>>> >>>> >>> >needs to
>>>> >>>> >>> >>> take the server CA cert from a memory blob instead from a
>>>> >file
>>>> >>>> >>> >(info.
>>>> >>>> >>> >>> ssl_ca_filepath). Can this be done?
>>>> >>>> >>>
>>>> >>>> >>> If you want to send a patch I'm happy to look at it.
>>>> >>>> >>>
>>>> >>>> >>> -Andy
>>>> >>>> >>>
>>>> >>>> >>> >>> Many thanks,
>>>> >>>> >>> >>> Lan
>>>> >>>> >>> >>>
>>>> >>>> >>> >>>
>>>> >>>> >>> >>>
>>>> >>>> >>> >>> On Sat, Dec 13, 2014 at 10:50 AM, Andy Green
>>>> ><andy at warmcat.com>
>>>> >>>> >>> >wrote:
>>>> >>>> >>> >>>
>>>> >>>> >>> >>>>
>>>> >>>> >>> >>>>
>>>> >>>> >>> >>>> On 13 December 2014 23:42:18 GMT+08:00, Duy Lan Nguyen <
>>>> >>>> >>> >>>> ndlan2k at gmail.com> wrote:
>>>> >>>> >>> >>>> >Hi Andy,
>>>> >>>> >>> >>>> >
>>>> >>>> >>> >>>> >Just to make sure, the sample codes for the strategy you
>>>> >>>> >>mentioned
>>>> >>>> >>> >>>> >(wait
>>>> >>>> >>> >>>> >for the WRITEABLE callback and the use a do { } while()
>>>> >to
>>>> >>>> >>stuff
>>>> >>>> >>> >the
>>>> >>>> >>> >>>> >pipe
>>>> >>>> >>> >>>> >with a frame of total length below "the number" until
>>>> >after
>>>> >>>> >the
>>>> >>>> >>> >last
>>>> >>>> >>> >>>> >one,
>>>> >>>> >>> >>>> >it has no more POLLOUT for you) are in test-server.c,
>>>> >>>> >>> >>>> >in LWS_CALLBACK_HTTP_WRITEABLE and
>>>> >>>> >>LWS_CALLBACK_SERVER_WRITEABLE,
>>>> >>>> >>> >where
>>>> >>>> >>> >>>> >you
>>>> >>>> >>> >>>> >used
>>>> >>>> >>> >>>> >
>>>> >>>> >>> >>>> >if (lws_partial_buffered(wsi) ||
>>>> >lws_send_pipe_choked(wsi)) {
>>>> >>>> >>> >>>> >libwebsocket_callback_on_writable(context, wsi);
>>>> >>>> >>> >>>> >break;
>>>> >>>> >>> >>>> >}
>>>> >>>> >>> >>>> >
>>>> >>>> >>> >>>> >to check if "it has no more POLLOUT for you". Is that
>>>> >right?
>>>> >>>> >>> >>>>
>>>> >>>> >>> >>>> Yes.  That way you can be sure you gave the kernel as much
>>>> >as
>>>> >>>> >it
>>>> >>>> >>> >could
>>>> >>>> >>> >>>> handle for that connection each time it had indicated you
>>>> >could
>>>> >>>> >>> >give the
>>>> >>>> >>> >>>> connection anything.
>>>> >>>> >>> >>>>
>>>> >>>> >>> >>>> -Andy
>>>> >>>> >>> >>>>
>>>> >>>> >>> >>>> >Many thanks,
>>>> >>>> >>> >>>> >Lan
>>>> >>>> >>> >>>> >
>>>> >>>> >>> >>>> >
>>>> >>>> >>> >>>> >
>>>> >>>> >>> >>>> >On Mon, Nov 24, 2014 at 2:45 PM, Andy Green
>>>> ><andy at warmcat.com>
>>>> >>>> >>> >wrote:
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> On 22 November 2014 16:42:32 GMT+08:00, Duy Lan Nguyen
>>>> >>>> >>> >>>> ><ndlan2k at gmail.com>
>>>> >>>> >>> >>>> >> wrote:
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> I am replying on the list.
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> >Hi Andy,
>>>> >>>> >>> >>>> >> >
>>>> >>>> >>> >>>> >> >I'll need to chop 1 MB message to atomic frames to
>>>> >send and
>>>> >>>> >>> >receive.
>>>> >>>> >>> >>>> >It
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> What is an "atomic frame"... if you mean a logical
>>>> >frame
>>>> >>>> >that
>>>> >>>> >>is
>>>> >>>> >>> >>>> >split
>>>> >>>> >>> >>>> >> into fragments of a larger logical message and sent
>>>> >like
>>>> >>>> >>that,
>>>> >>>> >>> >yes
>>>> >>>> >>> >>>> >you
>>>> >>>> >>> >>>> >> should do that.
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> >seems that the default atomic frame size is 4KB which
>>>> >is
>>>> >>>> >>> >>>> >> >LWS_MAX_SOCKET_IO_BUF.
>>>> >>>> >>> >>>> >> >
>>>> >>>> >>> >>>> >> >Why should it be 4 KB? Or what should be the best
>>>> >atomic
>>>> >>>> >>frame
>>>> >>>> >>> >size,
>>>> >>>> >>> >>>> >> >for
>>>> >>>> >>> >>>> >> >example for 1 MB message?
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> Well it depends on the server and the server load.
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> We avoid blocking by waiting for a POLLOUT indication.
>>>> >But
>>>> >>>> >>that
>>>> >>>> >>> >only
>>>> >>>> >>> >>>> >> tells us we could send "something", it doesn't say how
>>>> >much
>>>> >>>> >>we
>>>> >>>> >>> >could
>>>> >>>> >>> >>>> >send
>>>> >>>> >>> >>>> >> without blocking.  If we get POLLOUT and try to send
>>>> >10MB at
>>>> >>>> >>> >once, we
>>>> >>>> >>> >>>> >will
>>>> >>>> >>> >>>> >> get a partial send.
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> The reason is the kernel is not willing to allocate
>>>> >10MB
>>>> >>>> >>buffer
>>>> >>>> >>> >on
>>>> >>>> >>> >>>> >kernel
>>>> >>>> >>> >>>> >> side.  He did give us POLLOUT though, so he is willing
>>>> >to
>>>> >>>> >>take
>>>> >>>> >>> >>>> >something.
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> Empirically at least with linux on typical ethernet and
>>>> >WLAN
>>>> >>>> >>> >setups,
>>>> >>>> >>> >>>> >there
>>>> >>>> >>> >>>> >> is "a number" below which we don't get a POLLOUT
>>>> >indication.
>>>> >>>> >
>>>> >>>> >>It
>>>> >>>> >>> >>>> >seems to
>>>> >>>> >>> >>>> >> be around an ethernet frame size, eg, 1500 even if the
>>>> >box
>>>> >>>> >is
>>>> >>>> >>> >heavily
>>>> >>>> >>> >>>> >> loaded (at least, on my boxes).  If the box is less
>>>> >loaded,
>>>> >>>> >>it
>>>> >>>> >>> >will
>>>> >>>> >>> >>>> >accept
>>>> >>>> >>> >>>> >> more.
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> So one strategy is shown in the test server code, wait
>>>> >for
>>>> >>>> >>the
>>>> >>>> >>> >>>> >WRITEABLE
>>>> >>>> >>> >>>> >> callback and the use a do { } while() to stuff the pipe
>>>> >with
>>>> >>>> >>a
>>>> >>>> >>> >frame
>>>> >>>> >>> >>>> >of
>>>> >>>> >>> >>>> >> total length below "the number" until after the last
>>>> >one, it
>>>> >>>> >>has
>>>> >>>> >>> >no
>>>> >>>> >>> >>>> >more
>>>> >>>> >>> >>>> >> POLLOUT for you.  Then if you still have more, ask for
>>>> >>>> >>another
>>>> >>>> >>> >>>> >WRITEABLE.
>>>> >>>> >>> >>>> >> That way you should never block but also keep the
>>>> >kernel
>>>> >>>> >side
>>>> >>>> >>as
>>>> >>>> >>> >>>> >stuffed as
>>>> >>>> >>> >>>> >> possible, and take advantage of packet coalesce if
>>>> >possible.
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> Another strategy if you have a lot of memory is just
>>>> >rely on
>>>> >>>> >>the
>>>> >>>> >>> >>>> >partial
>>>> >>>> >>> >>>> >> send buffering in lws.  Do ask to send 1MB, when the
>>>> >kernel
>>>> >>>> >>just
>>>> >>>> >>> >>>> >takes 2K
>>>> >>>> >>> >>>> >> or whatever lws will copy the rest into a buffer and
>>>> >take
>>>> >>>> >>care
>>>> >>>> >>> >of
>>>> >>>> >>> >>>> >> prioritizing the send of the remaining buffered data
>>>> >before
>>>> >>>> >>that
>>>> >>>> >>> >>>> >connection
>>>> >>>> >>> >>>> >> will get another WRITABLE callback.  But 1) it's
>>>> >expensive
>>>> >>>> >>> >allocating
>>>> >>>> >>> >>>> >and
>>>> >>>> >>> >>>> >> copying around the buffer, and 2) when you use big
>>>> >frames
>>>> >>>> >you
>>>> >>>> >>> >block
>>>> >>>> >>> >>>> >the
>>>> >>>> >>> >>>> >> link for control frames like PING for long periods or
>>>> >from
>>>> >>>> >>> >sending
>>>> >>>> >>> >>>> >anything
>>>> >>>> >>> >>>> >> else, so your latency can be hurt.
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> -Andy
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >> >Many thanks,
>>>> >>>> >>> >>>> >> >Lan
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>> >>
>>>> >>>> >>> >>>>
>>>> >>>> >>> >>>>
>>>> >>>> >>> >>>
>>>> >>>> >>> >>
>>>> >>>> >>>
>>>> >>>> >>>
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> >>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20150411/f858dfc1/attachment-0001.html>


More information about the Libwebsockets mailing list