[Libwebsockets] Atomic Frame Size

Andy Green andy at warmcat.com
Sat Mar 28 13:34:40 CET 2015



On 28 March 2015 19:09:53 GMT+08:00, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:
>Hi Andy,
>
>Would it be possible to make this change?

Sure, send a patch.

The only windows this gets built on, an autobuilder, it builds already.

If your patch makes it build for you and doesn't break the autobuilder then fine...

-Andy

>We'd greatly appreciate!!!
>
>On Thu, Mar 26, 2015 at 10:17 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>wrote:
>
>> We need to use Visual Studio 2008 (VS 9) with libwebsockets to be
>> compatible with our Driver DK. It was ok with previous lws but the
>latest
>> lws doesn't compile as libwebsockets.h includes stdint.h that's not
>yet
>> available in VS 9. Could you please fix that so that lws will work
>with VS
>> 9?
>>
>> Thanks!!
>>
>> On Wed, Mar 25, 2015 at 1:43 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>wrote:
>>
>>> Awesome. Thanks
>>>
>>>
>>>
>>> On Tue, Mar 24, 2015 at 10:55 AM, Andy Green <andy at warmcat.com>
>wrote:
>>>
>>>>
>>>>
>>>> On 23 March 2015 22:28:45 GMT+08:00, Duy Lan Nguyen
><ndlan2k at gmail.com>
>>>> wrote:
>>>> >Yes, it's SSL connection
>>>>
>>>> I just pushed something that should solve it.
>>>>
>>>> -Andy
>>>>
>>>> >-----Original Message-----
>>>> >From: "Andy Green" <andy at warmcat.com>
>>>> >Sent: ‎3/‎23/‎2015 4:02 AM
>>>> >To: "Duy Lan Nguyen" <ndlan2k at gmail.com>
>>>> >Cc: "libwebsockets at ml.libwebsockets.org"
>>>> ><libwebsockets at ml.libwebsockets.org>
>>>> >Subject: Re: Atomic Frame Size
>>>> >
>>>> >
>>>> >
>>>> >On 23 March 2015 15:57:30 GMT+08:00, Duy Lan Nguyen
><ndlan2k at gmail.com>
>>>> >wrote:
>>>> >>Hi Andy,
>>>> >>
>>>> >>I still have the problem of libwebsocket_context_destroy()
>hanging,
>>>> >but
>>>> >>I
>>>> >>still don't understand its cause. It happens as follows in my
>Windows
>>>> >>machine.
>>>> >>
>>>> >>In a client-server connection, the server is suddenly shut down.
>The
>>>> >>client
>>>> >>doesn't realize it after a while, and finally closes down itself.
>When
>>>> >>the
>>>> >>client calls libwebsocket_context_destroy(), it hangs in the
>following
>>>> >>loop
>>>> >>inside libwebsocket_context_destroy()  as wsi is never NULL
>>>> >>
>>>> >>for (n = 0; n < context->fds_count; n++) {
>>>> >>struct libwebsocket *wsi =
>>>> >>context->lws_lookup[context->fds[n].fd];
>>>> >>if (!wsi)
>>>> >>continue;
>>>> >>libwebsocket_close_and_free_session(context,
>>>> >>wsi, LWS_CLOSE_STATUS_NOSTATUS /* no protocol close */);
>>>> >>n--;
>>>> >>}
>>>> >>
>>>> >>This happens as inside libwebsocket_close_and_free_session(), the
>code
>>>> >>keeps going into
>>>> >>
>>>> >>case WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE:
>>>> >>if (wsi->truncated_send_len) {
>>>> >>libwebsocket_callback_on_writable(context, wsi);
>>>> >>return;
>>>> >>}
>>>> >>
>>>> >>Did I do something wrong here or could you describe a solution to
>this
>>>> >
>>>> >Probably nothing more wrong than using windows.
>>>> >
>>>> >>hanging problem?
>>>> >
>>>> >I guess there's a case where he has something he wants to send
>before
>>>> >dying, but he can't send it.  Is it an SSL connection?
>>>> >
>>>> >-Andy
>>>> >
>>>> >>Thanks,
>>>> >>Lan
>>>> >>
>>>> >>
>>>> >>On Sun, Jan 25, 2015 at 8:13 PM, Andy Green <andy at warmcat.com>
>wrote:
>>>> >>
>>>> >>>
>>>> >>>
>>>> >>> On 11 January 2015 17:15:56 GMT+08:00, Duy Lan Nguyen
>>>> >><ndlan2k at gmail.com>
>>>> >>> wrote:
>>>> >>> >Hi Andy,
>>>> >>> >
>>>> >>> >I've implemented item 3). I can contribute it to lws if you
>want.
>>>> >>> >
>>>> >>> >Do you have any news on items 1) and 2)?
>>>> >>> >
>>>> >>> >Thanks,
>>>> >>> >Lan
>>>> >>> >
>>>> >>> >
>>>> >>> >On Tue, Dec 23, 2014 at 7:26 PM, Duy Lan Nguyen
><ndlan2k at gmail.com>
>>>> >>> >wrote:
>>>> >>> >
>>>> >>> >> In 1) and 2), I mean the server goes down suddenly
>>>> >>> >>
>>>> >>> >> On Tue, Dec 23, 2014 at 7:17 PM, Duy Lan Nguyen
>>>> >><ndlan2k at gmail.com>
>>>> >>> >wrote:
>>>> >>> >>
>>>> >>> >>> Hi Andy,
>>>> >>> >>>
>>>> >>> >>> I have some questions if you could help
>>>> >>> >>> 1) I'm trying to let client handle gracefully when the
>server
>>>> >>shuts
>>>> >>> >down.
>>>> >>> >>> With lots of traffic going back and forth (each of client
>and
>>>> >>server
>>>> >>> >has 2
>>>> >>> >>> threads, one for sending and one for receiving),
>>>> >>LWS_CALLBACK_CLOSED
>>>> >>> >>> sometimes isn't caught by polling, and its callback routine
>(in
>>>> >>the
>>>> >>> >>> switch-case) isn't called. Is it possible to make
>>>> >>> >LWS_CALLBACK_CLOSED
>>>> >>> >>> never be dropped?
>>>> >>>
>>>> >>> LWS never drops any close message AFAIK, but the OS may never
>become
>>>> >>aware
>>>> >>> that the connection is dead.  It doesn't know if the reason
>>>> >>everything is
>>>> >>> quiet is because everything is quiet, or because somewhere a
>network
>>>> >>cable
>>>> >>> fell out.
>>>> >>>
>>>> >>> You need to use keepalives of some sort (tcp, or PING / PONG in
>lws)
>>>> >>to
>>>> >>> detect that.
>>>> >>>
>>>> >>> >>> 2) And after the connection is already shutdown for a while
>>>> >>(client
>>>> >>> >>> doesn't know it as LWS_CALLBACK_CLOSED is dropped), calling
>>>> >>> >>> libwebsocket_context_destroy() is hanged. Is it possible to
>>>> >avoid
>>>> >>it
>>>> >>> >>> hanging?
>>>> >>>
>>>> >>> Sorry where does it hang in there?
>>>> >>>
>>>> >>> >>> 3) We want to build a single-file client application, so
>the app
>>>> >>> >needs to
>>>> >>> >>> take the server CA cert from a memory blob instead from a
>file
>>>> >>> >(info.
>>>> >>> >>> ssl_ca_filepath). Can this be done?
>>>> >>>
>>>> >>> If you want to send a patch I'm happy to look at it.
>>>> >>>
>>>> >>> -Andy
>>>> >>>
>>>> >>> >>> Many thanks,
>>>> >>> >>> Lan
>>>> >>> >>>
>>>> >>> >>>
>>>> >>> >>>
>>>> >>> >>> On Sat, Dec 13, 2014 at 10:50 AM, Andy Green
><andy at warmcat.com>
>>>> >>> >wrote:
>>>> >>> >>>
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>> On 13 December 2014 23:42:18 GMT+08:00, Duy Lan Nguyen <
>>>> >>> >>>> ndlan2k at gmail.com> wrote:
>>>> >>> >>>> >Hi Andy,
>>>> >>> >>>> >
>>>> >>> >>>> >Just to make sure, the sample codes for the strategy you
>>>> >>mentioned
>>>> >>> >>>> >(wait
>>>> >>> >>>> >for the WRITEABLE callback and the use a do { } while()
>to
>>>> >>stuff
>>>> >>> >the
>>>> >>> >>>> >pipe
>>>> >>> >>>> >with a frame of total length below "the number" until
>after
>>>> >the
>>>> >>> >last
>>>> >>> >>>> >one,
>>>> >>> >>>> >it has no more POLLOUT for you) are in test-server.c,
>>>> >>> >>>> >in LWS_CALLBACK_HTTP_WRITEABLE and
>>>> >>LWS_CALLBACK_SERVER_WRITEABLE,
>>>> >>> >where
>>>> >>> >>>> >you
>>>> >>> >>>> >used
>>>> >>> >>>> >
>>>> >>> >>>> >if (lws_partial_buffered(wsi) ||
>lws_send_pipe_choked(wsi)) {
>>>> >>> >>>> >libwebsocket_callback_on_writable(context, wsi);
>>>> >>> >>>> >break;
>>>> >>> >>>> >}
>>>> >>> >>>> >
>>>> >>> >>>> >to check if "it has no more POLLOUT for you". Is that
>right?
>>>> >>> >>>>
>>>> >>> >>>> Yes.  That way you can be sure you gave the kernel as much
>as
>>>> >it
>>>> >>> >could
>>>> >>> >>>> handle for that connection each time it had indicated you
>could
>>>> >>> >give the
>>>> >>> >>>> connection anything.
>>>> >>> >>>>
>>>> >>> >>>> -Andy
>>>> >>> >>>>
>>>> >>> >>>> >Many thanks,
>>>> >>> >>>> >Lan
>>>> >>> >>>> >
>>>> >>> >>>> >
>>>> >>> >>>> >
>>>> >>> >>>> >On Mon, Nov 24, 2014 at 2:45 PM, Andy Green
><andy at warmcat.com>
>>>> >>> >wrote:
>>>> >>> >>>> >>
>>>> >>> >>>> >>
>>>> >>> >>>> >>
>>>> >>> >>>> >> On 22 November 2014 16:42:32 GMT+08:00, Duy Lan Nguyen
>>>> >>> >>>> ><ndlan2k at gmail.com>
>>>> >>> >>>> >> wrote:
>>>> >>> >>>> >>
>>>> >>> >>>> >> I am replying on the list.
>>>> >>> >>>> >>
>>>> >>> >>>> >> >Hi Andy,
>>>> >>> >>>> >> >
>>>> >>> >>>> >> >I'll need to chop 1 MB message to atomic frames to
>send and
>>>> >>> >receive.
>>>> >>> >>>> >It
>>>> >>> >>>> >>
>>>> >>> >>>> >> What is an "atomic frame"... if you mean a logical
>frame
>>>> >that
>>>> >>is
>>>> >>> >>>> >split
>>>> >>> >>>> >> into fragments of a larger logical message and sent
>like
>>>> >>that,
>>>> >>> >yes
>>>> >>> >>>> >you
>>>> >>> >>>> >> should do that.
>>>> >>> >>>> >>
>>>> >>> >>>> >> >seems that the default atomic frame size is 4KB which
>is
>>>> >>> >>>> >> >LWS_MAX_SOCKET_IO_BUF.
>>>> >>> >>>> >> >
>>>> >>> >>>> >> >Why should it be 4 KB? Or what should be the best
>atomic
>>>> >>frame
>>>> >>> >size,
>>>> >>> >>>> >> >for
>>>> >>> >>>> >> >example for 1 MB message?
>>>> >>> >>>> >>
>>>> >>> >>>> >> Well it depends on the server and the server load.
>>>> >>> >>>> >>
>>>> >>> >>>> >> We avoid blocking by waiting for a POLLOUT indication. 
>But
>>>> >>that
>>>> >>> >only
>>>> >>> >>>> >> tells us we could send "something", it doesn't say how
>much
>>>> >>we
>>>> >>> >could
>>>> >>> >>>> >send
>>>> >>> >>>> >> without blocking.  If we get POLLOUT and try to send
>10MB at
>>>> >>> >once, we
>>>> >>> >>>> >will
>>>> >>> >>>> >> get a partial send.
>>>> >>> >>>> >>
>>>> >>> >>>> >> The reason is the kernel is not willing to allocate
>10MB
>>>> >>buffer
>>>> >>> >on
>>>> >>> >>>> >kernel
>>>> >>> >>>> >> side.  He did give us POLLOUT though, so he is willing
>to
>>>> >>take
>>>> >>> >>>> >something.
>>>> >>> >>>> >>
>>>> >>> >>>> >> Empirically at least with linux on typical ethernet and
>WLAN
>>>> >>> >setups,
>>>> >>> >>>> >there
>>>> >>> >>>> >> is "a number" below which we don't get a POLLOUT
>indication.
>>>> >
>>>> >>It
>>>> >>> >>>> >seems to
>>>> >>> >>>> >> be around an ethernet frame size, eg, 1500 even if the
>box
>>>> >is
>>>> >>> >heavily
>>>> >>> >>>> >> loaded (at least, on my boxes).  If the box is less
>loaded,
>>>> >>it
>>>> >>> >will
>>>> >>> >>>> >accept
>>>> >>> >>>> >> more.
>>>> >>> >>>> >>
>>>> >>> >>>> >> So one strategy is shown in the test server code, wait
>for
>>>> >>the
>>>> >>> >>>> >WRITEABLE
>>>> >>> >>>> >> callback and the use a do { } while() to stuff the pipe
>with
>>>> >>a
>>>> >>> >frame
>>>> >>> >>>> >of
>>>> >>> >>>> >> total length below "the number" until after the last
>one, it
>>>> >>has
>>>> >>> >no
>>>> >>> >>>> >more
>>>> >>> >>>> >> POLLOUT for you.  Then if you still have more, ask for
>>>> >>another
>>>> >>> >>>> >WRITEABLE.
>>>> >>> >>>> >> That way you should never block but also keep the
>kernel
>>>> >side
>>>> >>as
>>>> >>> >>>> >stuffed as
>>>> >>> >>>> >> possible, and take advantage of packet coalesce if
>possible.
>>>> >>> >>>> >>
>>>> >>> >>>> >> Another strategy if you have a lot of memory is just
>rely on
>>>> >>the
>>>> >>> >>>> >partial
>>>> >>> >>>> >> send buffering in lws.  Do ask to send 1MB, when the
>kernel
>>>> >>just
>>>> >>> >>>> >takes 2K
>>>> >>> >>>> >> or whatever lws will copy the rest into a buffer and
>take
>>>> >>care
>>>> >>> >of
>>>> >>> >>>> >> prioritizing the send of the remaining buffered data
>before
>>>> >>that
>>>> >>> >>>> >connection
>>>> >>> >>>> >> will get another WRITABLE callback.  But 1) it's
>expensive
>>>> >>> >allocating
>>>> >>> >>>> >and
>>>> >>> >>>> >> copying around the buffer, and 2) when you use big
>frames
>>>> >you
>>>> >>> >block
>>>> >>> >>>> >the
>>>> >>> >>>> >> link for control frames like PING for long periods or
>from
>>>> >>> >sending
>>>> >>> >>>> >anything
>>>> >>> >>>> >> else, so your latency can be hurt.
>>>> >>> >>>> >>
>>>> >>> >>>> >> -Andy
>>>> >>> >>>> >>
>>>> >>> >>>> >> >Many thanks,
>>>> >>> >>>> >> >Lan
>>>> >>> >>>> >>
>>>> >>> >>>> >>
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>
>>>> >>> >>
>>>> >>>
>>>> >>>
>>>>
>>>>
>>>
>>




More information about the Libwebsockets mailing list