[Libwebsockets] Atomic Frame Size

Andy Green andy at warmcat.com
Mon Mar 23 12:02:06 CET 2015



On 23 March 2015 15:57:30 GMT+08:00, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:
>Hi Andy,
>
>I still have the problem of libwebsocket_context_destroy() hanging, but
>I
>still don't understand its cause. It happens as follows in my Windows
>machine.
>
>In a client-server connection, the server is suddenly shut down. The
>client
>doesn't realize it after a while, and finally closes down itself. When
>the
>client calls libwebsocket_context_destroy(), it hangs in the following
>loop
>inside libwebsocket_context_destroy()  as wsi is never NULL
>
>for (n = 0; n < context->fds_count; n++) {
>struct libwebsocket *wsi =
>context->lws_lookup[context->fds[n].fd];
>if (!wsi)
>continue;
>libwebsocket_close_and_free_session(context,
>wsi, LWS_CLOSE_STATUS_NOSTATUS /* no protocol close */);
>n--;
>}
>
>This happens as inside libwebsocket_close_and_free_session(), the code
>keeps going into
>
>case WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE:
>if (wsi->truncated_send_len) {
>libwebsocket_callback_on_writable(context, wsi);
>return;
>}
>
>Did I do something wrong here or could you describe a solution to this

Probably nothing more wrong than using windows.

>hanging problem?

I guess there's a case where he has something he wants to send before dying, but he can't send it.  Is it an SSL connection?

-Andy

>Thanks,
>Lan
>
>
>On Sun, Jan 25, 2015 at 8:13 PM, Andy Green <andy at warmcat.com> wrote:
>
>>
>>
>> On 11 January 2015 17:15:56 GMT+08:00, Duy Lan Nguyen
><ndlan2k at gmail.com>
>> wrote:
>> >Hi Andy,
>> >
>> >I've implemented item 3). I can contribute it to lws if you want.
>> >
>> >Do you have any news on items 1) and 2)?
>> >
>> >Thanks,
>> >Lan
>> >
>> >
>> >On Tue, Dec 23, 2014 at 7:26 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>> >wrote:
>> >
>> >> In 1) and 2), I mean the server goes down suddenly
>> >>
>> >> On Tue, Dec 23, 2014 at 7:17 PM, Duy Lan Nguyen
><ndlan2k at gmail.com>
>> >wrote:
>> >>
>> >>> Hi Andy,
>> >>>
>> >>> I have some questions if you could help
>> >>> 1) I'm trying to let client handle gracefully when the server
>shuts
>> >down.
>> >>> With lots of traffic going back and forth (each of client and
>server
>> >has 2
>> >>> threads, one for sending and one for receiving),
>LWS_CALLBACK_CLOSED
>> >>> sometimes isn't caught by polling, and its callback routine (in
>the
>> >>> switch-case) isn't called. Is it possible to make
>> >LWS_CALLBACK_CLOSED
>> >>> never be dropped?
>>
>> LWS never drops any close message AFAIK, but the OS may never become
>aware
>> that the connection is dead.  It doesn't know if the reason
>everything is
>> quiet is because everything is quiet, or because somewhere a network
>cable
>> fell out.
>>
>> You need to use keepalives of some sort (tcp, or PING / PONG in lws)
>to
>> detect that.
>>
>> >>> 2) And after the connection is already shutdown for a while
>(client
>> >>> doesn't know it as LWS_CALLBACK_CLOSED is dropped), calling
>> >>> libwebsocket_context_destroy() is hanged. Is it possible to avoid
>it
>> >>> hanging?
>>
>> Sorry where does it hang in there?
>>
>> >>> 3) We want to build a single-file client application, so the app
>> >needs to
>> >>> take the server CA cert from a memory blob instead from a file
>> >(info.
>> >>> ssl_ca_filepath). Can this be done?
>>
>> If you want to send a patch I'm happy to look at it.
>>
>> -Andy
>>
>> >>> Many thanks,
>> >>> Lan
>> >>>
>> >>>
>> >>>
>> >>> On Sat, Dec 13, 2014 at 10:50 AM, Andy Green <andy at warmcat.com>
>> >wrote:
>> >>>
>> >>>>
>> >>>>
>> >>>> On 13 December 2014 23:42:18 GMT+08:00, Duy Lan Nguyen <
>> >>>> ndlan2k at gmail.com> wrote:
>> >>>> >Hi Andy,
>> >>>> >
>> >>>> >Just to make sure, the sample codes for the strategy you
>mentioned
>> >>>> >(wait
>> >>>> >for the WRITEABLE callback and the use a do { } while() to
>stuff
>> >the
>> >>>> >pipe
>> >>>> >with a frame of total length below "the number" until after the
>> >last
>> >>>> >one,
>> >>>> >it has no more POLLOUT for you) are in test-server.c,
>> >>>> >in LWS_CALLBACK_HTTP_WRITEABLE and
>LWS_CALLBACK_SERVER_WRITEABLE,
>> >where
>> >>>> >you
>> >>>> >used
>> >>>> >
>> >>>> >if (lws_partial_buffered(wsi) || lws_send_pipe_choked(wsi)) {
>> >>>> >libwebsocket_callback_on_writable(context, wsi);
>> >>>> >break;
>> >>>> >}
>> >>>> >
>> >>>> >to check if "it has no more POLLOUT for you". Is that right?
>> >>>>
>> >>>> Yes.  That way you can be sure you gave the kernel as much as it
>> >could
>> >>>> handle for that connection each time it had indicated you could
>> >give the
>> >>>> connection anything.
>> >>>>
>> >>>> -Andy
>> >>>>
>> >>>> >Many thanks,
>> >>>> >Lan
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >On Mon, Nov 24, 2014 at 2:45 PM, Andy Green <andy at warmcat.com>
>> >wrote:
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> On 22 November 2014 16:42:32 GMT+08:00, Duy Lan Nguyen
>> >>>> ><ndlan2k at gmail.com>
>> >>>> >> wrote:
>> >>>> >>
>> >>>> >> I am replying on the list.
>> >>>> >>
>> >>>> >> >Hi Andy,
>> >>>> >> >
>> >>>> >> >I'll need to chop 1 MB message to atomic frames to send and
>> >receive.
>> >>>> >It
>> >>>> >>
>> >>>> >> What is an "atomic frame"... if you mean a logical frame that
>is
>> >>>> >split
>> >>>> >> into fragments of a larger logical message and sent like
>that,
>> >yes
>> >>>> >you
>> >>>> >> should do that.
>> >>>> >>
>> >>>> >> >seems that the default atomic frame size is 4KB which is
>> >>>> >> >LWS_MAX_SOCKET_IO_BUF.
>> >>>> >> >
>> >>>> >> >Why should it be 4 KB? Or what should be the best atomic
>frame
>> >size,
>> >>>> >> >for
>> >>>> >> >example for 1 MB message?
>> >>>> >>
>> >>>> >> Well it depends on the server and the server load.
>> >>>> >>
>> >>>> >> We avoid blocking by waiting for a POLLOUT indication.  But
>that
>> >only
>> >>>> >> tells us we could send "something", it doesn't say how much
>we
>> >could
>> >>>> >send
>> >>>> >> without blocking.  If we get POLLOUT and try to send 10MB at
>> >once, we
>> >>>> >will
>> >>>> >> get a partial send.
>> >>>> >>
>> >>>> >> The reason is the kernel is not willing to allocate 10MB
>buffer
>> >on
>> >>>> >kernel
>> >>>> >> side.  He did give us POLLOUT though, so he is willing to
>take
>> >>>> >something.
>> >>>> >>
>> >>>> >> Empirically at least with linux on typical ethernet and WLAN
>> >setups,
>> >>>> >there
>> >>>> >> is "a number" below which we don't get a POLLOUT indication. 
>It
>> >>>> >seems to
>> >>>> >> be around an ethernet frame size, eg, 1500 even if the box is
>> >heavily
>> >>>> >> loaded (at least, on my boxes).  If the box is less loaded,
>it
>> >will
>> >>>> >accept
>> >>>> >> more.
>> >>>> >>
>> >>>> >> So one strategy is shown in the test server code, wait for
>the
>> >>>> >WRITEABLE
>> >>>> >> callback and the use a do { } while() to stuff the pipe with
>a
>> >frame
>> >>>> >of
>> >>>> >> total length below "the number" until after the last one, it
>has
>> >no
>> >>>> >more
>> >>>> >> POLLOUT for you.  Then if you still have more, ask for
>another
>> >>>> >WRITEABLE.
>> >>>> >> That way you should never block but also keep the kernel side
>as
>> >>>> >stuffed as
>> >>>> >> possible, and take advantage of packet coalesce if possible.
>> >>>> >>
>> >>>> >> Another strategy if you have a lot of memory is just rely on
>the
>> >>>> >partial
>> >>>> >> send buffering in lws.  Do ask to send 1MB, when the kernel
>just
>> >>>> >takes 2K
>> >>>> >> or whatever lws will copy the rest into a buffer and take
>care
>> >of
>> >>>> >> prioritizing the send of the remaining buffered data before
>that
>> >>>> >connection
>> >>>> >> will get another WRITABLE callback.  But 1) it's expensive
>> >allocating
>> >>>> >and
>> >>>> >> copying around the buffer, and 2) when you use big frames you
>> >block
>> >>>> >the
>> >>>> >> link for control frames like PING for long periods or from
>> >sending
>> >>>> >anything
>> >>>> >> else, so your latency can be hurt.
>> >>>> >>
>> >>>> >> -Andy
>> >>>> >>
>> >>>> >> >Many thanks,
>> >>>> >> >Lan
>> >>>> >>
>> >>>> >>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>




More information about the Libwebsockets mailing list