[Libwebsockets] Atomic Frame Size

Andy Green andy at warmcat.com
Wed Apr 1 21:04:29 CEST 2015



On 1 April 2015 23:50:32 GMT+08:00, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:
>Turn out that just removing #include stdint.h is enough and lws still
>works
>fine.

Okay... we can try it on the autobuilder which will check if that's true with more Cmake options.

>I made another change to allow using cert blobs instead of using cert
>files, with cyassl. I can provide that as well?

I am not sure what 'cert blob' means, it's pkcs12?

Sure send the patch.

>Should I email changes to you or push it to git?

If you use github, push it to your github clone and send a pull request.

Or post the patches here.

-Andy

>Thanks
>
>On Sat, Mar 28, 2015 at 5:34 AM, Andy Green <andy at warmcat.com> wrote:
>
>>
>>
>> On 28 March 2015 19:09:53 GMT+08:00, Duy Lan Nguyen
><ndlan2k at gmail.com>
>> wrote:
>> >Hi Andy,
>> >
>> >Would it be possible to make this change?
>>
>> Sure, send a patch.
>>
>> The only windows this gets built on, an autobuilder, it builds
>already.
>>
>> If your patch makes it build for you and doesn't break the
>autobuilder
>> then fine...
>>
>> -Andy
>>
>> >We'd greatly appreciate!!!
>> >
>> >On Thu, Mar 26, 2015 at 10:17 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>> >wrote:
>> >
>> >> We need to use Visual Studio 2008 (VS 9) with libwebsockets to be
>> >> compatible with our Driver DK. It was ok with previous lws but the
>> >latest
>> >> lws doesn't compile as libwebsockets.h includes stdint.h that's
>not
>> >yet
>> >> available in VS 9. Could you please fix that so that lws will work
>> >with VS
>> >> 9?
>> >>
>> >> Thanks!!
>> >>
>> >> On Wed, Mar 25, 2015 at 1:43 PM, Duy Lan Nguyen
><ndlan2k at gmail.com>
>> >wrote:
>> >>
>> >>> Awesome. Thanks
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Mar 24, 2015 at 10:55 AM, Andy Green <andy at warmcat.com>
>> >wrote:
>> >>>
>> >>>>
>> >>>>
>> >>>> On 23 March 2015 22:28:45 GMT+08:00, Duy Lan Nguyen
>> ><ndlan2k at gmail.com>
>> >>>> wrote:
>> >>>> >Yes, it's SSL connection
>> >>>>
>> >>>> I just pushed something that should solve it.
>> >>>>
>> >>>> -Andy
>> >>>>
>> >>>> >-----Original Message-----
>> >>>> >From: "Andy Green" <andy at warmcat.com>
>> >>>> >Sent: ‎3/‎23/‎2015 4:02 AM
>> >>>> >To: "Duy Lan Nguyen" <ndlan2k at gmail.com>
>> >>>> >Cc: "libwebsockets at ml.libwebsockets.org"
>> >>>> ><libwebsockets at ml.libwebsockets.org>
>> >>>> >Subject: Re: Atomic Frame Size
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >On 23 March 2015 15:57:30 GMT+08:00, Duy Lan Nguyen
>> ><ndlan2k at gmail.com>
>> >>>> >wrote:
>> >>>> >>Hi Andy,
>> >>>> >>
>> >>>> >>I still have the problem of libwebsocket_context_destroy()
>> >hanging,
>> >>>> >but
>> >>>> >>I
>> >>>> >>still don't understand its cause. It happens as follows in my
>> >Windows
>> >>>> >>machine.
>> >>>> >>
>> >>>> >>In a client-server connection, the server is suddenly shut
>down.
>> >The
>> >>>> >>client
>> >>>> >>doesn't realize it after a while, and finally closes down
>itself.
>> >When
>> >>>> >>the
>> >>>> >>client calls libwebsocket_context_destroy(), it hangs in the
>> >following
>> >>>> >>loop
>> >>>> >>inside libwebsocket_context_destroy()  as wsi is never NULL
>> >>>> >>
>> >>>> >>for (n = 0; n < context->fds_count; n++) {
>> >>>> >>struct libwebsocket *wsi =
>> >>>> >>context->lws_lookup[context->fds[n].fd];
>> >>>> >>if (!wsi)
>> >>>> >>continue;
>> >>>> >>libwebsocket_close_and_free_session(context,
>> >>>> >>wsi, LWS_CLOSE_STATUS_NOSTATUS /* no protocol close */);
>> >>>> >>n--;
>> >>>> >>}
>> >>>> >>
>> >>>> >>This happens as inside libwebsocket_close_and_free_session(),
>the
>> >code
>> >>>> >>keeps going into
>> >>>> >>
>> >>>> >>case WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE:
>> >>>> >>if (wsi->truncated_send_len) {
>> >>>> >>libwebsocket_callback_on_writable(context, wsi);
>> >>>> >>return;
>> >>>> >>}
>> >>>> >>
>> >>>> >>Did I do something wrong here or could you describe a solution
>to
>> >this
>> >>>> >
>> >>>> >Probably nothing more wrong than using windows.
>> >>>> >
>> >>>> >>hanging problem?
>> >>>> >
>> >>>> >I guess there's a case where he has something he wants to send
>> >before
>> >>>> >dying, but he can't send it.  Is it an SSL connection?
>> >>>> >
>> >>>> >-Andy
>> >>>> >
>> >>>> >>Thanks,
>> >>>> >>Lan
>> >>>> >>
>> >>>> >>
>> >>>> >>On Sun, Jan 25, 2015 at 8:13 PM, Andy Green <andy at warmcat.com>
>> >wrote:
>> >>>> >>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>> On 11 January 2015 17:15:56 GMT+08:00, Duy Lan Nguyen
>> >>>> >><ndlan2k at gmail.com>
>> >>>> >>> wrote:
>> >>>> >>> >Hi Andy,
>> >>>> >>> >
>> >>>> >>> >I've implemented item 3). I can contribute it to lws if you
>> >want.
>> >>>> >>> >
>> >>>> >>> >Do you have any news on items 1) and 2)?
>> >>>> >>> >
>> >>>> >>> >Thanks,
>> >>>> >>> >Lan
>> >>>> >>> >
>> >>>> >>> >
>> >>>> >>> >On Tue, Dec 23, 2014 at 7:26 PM, Duy Lan Nguyen
>> ><ndlan2k at gmail.com>
>> >>>> >>> >wrote:
>> >>>> >>> >
>> >>>> >>> >> In 1) and 2), I mean the server goes down suddenly
>> >>>> >>> >>
>> >>>> >>> >> On Tue, Dec 23, 2014 at 7:17 PM, Duy Lan Nguyen
>> >>>> >><ndlan2k at gmail.com>
>> >>>> >>> >wrote:
>> >>>> >>> >>
>> >>>> >>> >>> Hi Andy,
>> >>>> >>> >>>
>> >>>> >>> >>> I have some questions if you could help
>> >>>> >>> >>> 1) I'm trying to let client handle gracefully when the
>> >server
>> >>>> >>shuts
>> >>>> >>> >down.
>> >>>> >>> >>> With lots of traffic going back and forth (each of
>client
>> >and
>> >>>> >>server
>> >>>> >>> >has 2
>> >>>> >>> >>> threads, one for sending and one for receiving),
>> >>>> >>LWS_CALLBACK_CLOSED
>> >>>> >>> >>> sometimes isn't caught by polling, and its callback
>routine
>> >(in
>> >>>> >>the
>> >>>> >>> >>> switch-case) isn't called. Is it possible to make
>> >>>> >>> >LWS_CALLBACK_CLOSED
>> >>>> >>> >>> never be dropped?
>> >>>> >>>
>> >>>> >>> LWS never drops any close message AFAIK, but the OS may
>never
>> >become
>> >>>> >>aware
>> >>>> >>> that the connection is dead.  It doesn't know if the reason
>> >>>> >>everything is
>> >>>> >>> quiet is because everything is quiet, or because somewhere a
>> >network
>> >>>> >>cable
>> >>>> >>> fell out.
>> >>>> >>>
>> >>>> >>> You need to use keepalives of some sort (tcp, or PING / PONG
>in
>> >lws)
>> >>>> >>to
>> >>>> >>> detect that.
>> >>>> >>>
>> >>>> >>> >>> 2) And after the connection is already shutdown for a
>while
>> >>>> >>(client
>> >>>> >>> >>> doesn't know it as LWS_CALLBACK_CLOSED is dropped),
>calling
>> >>>> >>> >>> libwebsocket_context_destroy() is hanged. Is it possible
>to
>> >>>> >avoid
>> >>>> >>it
>> >>>> >>> >>> hanging?
>> >>>> >>>
>> >>>> >>> Sorry where does it hang in there?
>> >>>> >>>
>> >>>> >>> >>> 3) We want to build a single-file client application, so
>> >the app
>> >>>> >>> >needs to
>> >>>> >>> >>> take the server CA cert from a memory blob instead from
>a
>> >file
>> >>>> >>> >(info.
>> >>>> >>> >>> ssl_ca_filepath). Can this be done?
>> >>>> >>>
>> >>>> >>> If you want to send a patch I'm happy to look at it.
>> >>>> >>>
>> >>>> >>> -Andy
>> >>>> >>>
>> >>>> >>> >>> Many thanks,
>> >>>> >>> >>> Lan
>> >>>> >>> >>>
>> >>>> >>> >>>
>> >>>> >>> >>>
>> >>>> >>> >>> On Sat, Dec 13, 2014 at 10:50 AM, Andy Green
>> ><andy at warmcat.com>
>> >>>> >>> >wrote:
>> >>>> >>> >>>
>> >>>> >>> >>>>
>> >>>> >>> >>>>
>> >>>> >>> >>>> On 13 December 2014 23:42:18 GMT+08:00, Duy Lan Nguyen
><
>> >>>> >>> >>>> ndlan2k at gmail.com> wrote:
>> >>>> >>> >>>> >Hi Andy,
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >Just to make sure, the sample codes for the strategy
>you
>> >>>> >>mentioned
>> >>>> >>> >>>> >(wait
>> >>>> >>> >>>> >for the WRITEABLE callback and the use a do { }
>while()
>> >to
>> >>>> >>stuff
>> >>>> >>> >the
>> >>>> >>> >>>> >pipe
>> >>>> >>> >>>> >with a frame of total length below "the number" until
>> >after
>> >>>> >the
>> >>>> >>> >last
>> >>>> >>> >>>> >one,
>> >>>> >>> >>>> >it has no more POLLOUT for you) are in test-server.c,
>> >>>> >>> >>>> >in LWS_CALLBACK_HTTP_WRITEABLE and
>> >>>> >>LWS_CALLBACK_SERVER_WRITEABLE,
>> >>>> >>> >where
>> >>>> >>> >>>> >you
>> >>>> >>> >>>> >used
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >if (lws_partial_buffered(wsi) ||
>> >lws_send_pipe_choked(wsi)) {
>> >>>> >>> >>>> >libwebsocket_callback_on_writable(context, wsi);
>> >>>> >>> >>>> >break;
>> >>>> >>> >>>> >}
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >to check if "it has no more POLLOUT for you". Is that
>> >right?
>> >>>> >>> >>>>
>> >>>> >>> >>>> Yes.  That way you can be sure you gave the kernel as
>much
>> >as
>> >>>> >it
>> >>>> >>> >could
>> >>>> >>> >>>> handle for that connection each time it had indicated
>you
>> >could
>> >>>> >>> >give the
>> >>>> >>> >>>> connection anything.
>> >>>> >>> >>>>
>> >>>> >>> >>>> -Andy
>> >>>> >>> >>>>
>> >>>> >>> >>>> >Many thanks,
>> >>>> >>> >>>> >Lan
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >On Mon, Nov 24, 2014 at 2:45 PM, Andy Green
>> ><andy at warmcat.com>
>> >>>> >>> >wrote:
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> On 22 November 2014 16:42:32 GMT+08:00, Duy Lan
>Nguyen
>> >>>> >>> >>>> ><ndlan2k at gmail.com>
>> >>>> >>> >>>> >> wrote:
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> I am replying on the list.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> >Hi Andy,
>> >>>> >>> >>>> >> >
>> >>>> >>> >>>> >> >I'll need to chop 1 MB message to atomic frames to
>> >send and
>> >>>> >>> >receive.
>> >>>> >>> >>>> >It
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> What is an "atomic frame"... if you mean a logical
>> >frame
>> >>>> >that
>> >>>> >>is
>> >>>> >>> >>>> >split
>> >>>> >>> >>>> >> into fragments of a larger logical message and sent
>> >like
>> >>>> >>that,
>> >>>> >>> >yes
>> >>>> >>> >>>> >you
>> >>>> >>> >>>> >> should do that.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> >seems that the default atomic frame size is 4KB
>which
>> >is
>> >>>> >>> >>>> >> >LWS_MAX_SOCKET_IO_BUF.
>> >>>> >>> >>>> >> >
>> >>>> >>> >>>> >> >Why should it be 4 KB? Or what should be the best
>> >atomic
>> >>>> >>frame
>> >>>> >>> >size,
>> >>>> >>> >>>> >> >for
>> >>>> >>> >>>> >> >example for 1 MB message?
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> Well it depends on the server and the server load.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> We avoid blocking by waiting for a POLLOUT
>indication.
>> >But
>> >>>> >>that
>> >>>> >>> >only
>> >>>> >>> >>>> >> tells us we could send "something", it doesn't say
>how
>> >much
>> >>>> >>we
>> >>>> >>> >could
>> >>>> >>> >>>> >send
>> >>>> >>> >>>> >> without blocking.  If we get POLLOUT and try to send
>> >10MB at
>> >>>> >>> >once, we
>> >>>> >>> >>>> >will
>> >>>> >>> >>>> >> get a partial send.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> The reason is the kernel is not willing to allocate
>> >10MB
>> >>>> >>buffer
>> >>>> >>> >on
>> >>>> >>> >>>> >kernel
>> >>>> >>> >>>> >> side.  He did give us POLLOUT though, so he is
>willing
>> >to
>> >>>> >>take
>> >>>> >>> >>>> >something.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> Empirically at least with linux on typical ethernet
>and
>> >WLAN
>> >>>> >>> >setups,
>> >>>> >>> >>>> >there
>> >>>> >>> >>>> >> is "a number" below which we don't get a POLLOUT
>> >indication.
>> >>>> >
>> >>>> >>It
>> >>>> >>> >>>> >seems to
>> >>>> >>> >>>> >> be around an ethernet frame size, eg, 1500 even if
>the
>> >box
>> >>>> >is
>> >>>> >>> >heavily
>> >>>> >>> >>>> >> loaded (at least, on my boxes).  If the box is less
>> >loaded,
>> >>>> >>it
>> >>>> >>> >will
>> >>>> >>> >>>> >accept
>> >>>> >>> >>>> >> more.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> So one strategy is shown in the test server code,
>wait
>> >for
>> >>>> >>the
>> >>>> >>> >>>> >WRITEABLE
>> >>>> >>> >>>> >> callback and the use a do { } while() to stuff the
>pipe
>> >with
>> >>>> >>a
>> >>>> >>> >frame
>> >>>> >>> >>>> >of
>> >>>> >>> >>>> >> total length below "the number" until after the last
>> >one, it
>> >>>> >>has
>> >>>> >>> >no
>> >>>> >>> >>>> >more
>> >>>> >>> >>>> >> POLLOUT for you.  Then if you still have more, ask
>for
>> >>>> >>another
>> >>>> >>> >>>> >WRITEABLE.
>> >>>> >>> >>>> >> That way you should never block but also keep the
>> >kernel
>> >>>> >side
>> >>>> >>as
>> >>>> >>> >>>> >stuffed as
>> >>>> >>> >>>> >> possible, and take advantage of packet coalesce if
>> >possible.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> Another strategy if you have a lot of memory is just
>> >rely on
>> >>>> >>the
>> >>>> >>> >>>> >partial
>> >>>> >>> >>>> >> send buffering in lws.  Do ask to send 1MB, when the
>> >kernel
>> >>>> >>just
>> >>>> >>> >>>> >takes 2K
>> >>>> >>> >>>> >> or whatever lws will copy the rest into a buffer and
>> >take
>> >>>> >>care
>> >>>> >>> >of
>> >>>> >>> >>>> >> prioritizing the send of the remaining buffered data
>> >before
>> >>>> >>that
>> >>>> >>> >>>> >connection
>> >>>> >>> >>>> >> will get another WRITABLE callback.  But 1) it's
>> >expensive
>> >>>> >>> >allocating
>> >>>> >>> >>>> >and
>> >>>> >>> >>>> >> copying around the buffer, and 2) when you use big
>> >frames
>> >>>> >you
>> >>>> >>> >block
>> >>>> >>> >>>> >the
>> >>>> >>> >>>> >> link for control frames like PING for long periods
>or
>> >from
>> >>>> >>> >sending
>> >>>> >>> >>>> >anything
>> >>>> >>> >>>> >> else, so your latency can be hurt.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> -Andy
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> >Many thanks,
>> >>>> >>> >>>> >> >Lan
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >>
>> >>>> >>> >>>>
>> >>>> >>> >>>>
>> >>>> >>> >>>
>> >>>> >>> >>
>> >>>> >>>
>> >>>> >>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>




More information about the Libwebsockets mailing list