[Libwebsockets] Atomic Frame Size

Duy Lan Nguyen ndlan2k at gmail.com
Sat Apr 11 23:36:50 CEST 2015


Playing with lws some more, I think I see why LWS_CALLBACK_CLOSED is never
reached in that case. In libwebsocket_close_and_free_session, I see
if (wsi->truncated_send_len) { lwsl_info("wsi %p entering
WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE\n", wsi); wsi->state =
WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE; libwebsocket_set_timeout(wsi,
PENDING_FLUSH_STORED_SEND_BEFORE_CLOSE, 5); return; }
that means if there is still something to send, it will try to flush it
rather continuing in libwebsocket_close_and_free_session and eventually
calling LWS_CALLBACK_CLOSED.

Is it possible to do that code block later with more checking to make sure
that flushing is worth trying. otherwise just continuing to
LWS_CALLBACK_CLOSED?
Or if you have other suggestions?

Many thanks,
Lan

On Sat, Apr 11, 2015 at 10:53 AM, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:

> Hi Andy,
>
> We test how lws client handles server's sudden shutdown and observe the
> following.
>
> We use the method in call_back you advised before to keep the kernel
> stuffed when sending (do while loop with lws_partial_buffered and
> lws_send_pipe_choked).
>
>     case LWS_CALLBACK_CLIENT_WRITEABLE:
> do {
> nTemp1 = (*ppSocketInfo)->m_SendInfo.m_cbSendData -
> (*ppSocketInfo)->m_SendInfo.m_nBytesSent;
>
> nTemp2 = (nTemp1 > ATOM_DATA_SIZE) ? ATOM_DATA_SIZE : nTemp1;
>
> if
> (memcpy_s(&(*ppSocketInfo)->m_SendInfo.m_rgbSendBuffer[LWS_SEND_BUFFER_PRE_PADDING],
> ATOM_DATA_SIZE,
> (*ppSocketInfo)->m_SendInfo.m_pbSendData +
> (*ppSocketInfo)->m_SendInfo.m_nBytesSent,
> nTemp2))
> {
> (*ppSocketInfo)->m_SendInfo.m_retSend = FAIL;
> goto exit_send;
> }
>
> n = libwebsocket_write(wsi,
> &(*ppSocketInfo)->m_SendInfo.m_rgbSendBuffer[LWS_SEND_BUFFER_PRE_PADDING],
> nTemp2,
> nOpts | LWS_WRITE_BINARY);
>
> if (n < 0)
> {
> (*ppSocketInfo)->m_SendInfo.m_retSend = FAIL;
> goto exit_send;
> }
>  (*ppSocketInfo)->m_SendInfo.m_nBytesSent += n;
>
> if ((*ppSocketInfo)->m_SendInfo.m_nBytesSent >=
> (*ppSocketInfo)->m_SendInfo.m_cbSendData)
> {
> goto exit_send;
> }
>
> }
>
> if (lws_partial_buffered(wsi))
> break;
> } while (!lws_send_pipe_choked(wsi));
>
> In case sending data is smaller (such as 100 KB), I guess this loop
> completes faster. So when the server shuts down suddenly, the client side
> immediately goes into case LWS_CALLBACK_CLOSED and stops right away.
>
> In case sending data is big (such as 10 MB), the kernel seems never get
> stuffed (the machine is pretty powerful), and the whole 10 MB just uses the
> loop of a single case LWS_CALLBACK_CLIENT_WRITEABLE and it takes a bit
> longer. So when the server shuts down suddenly, although the loop also
> stops right away, the case LWS_CALLBACK_CLOSED is never reached. And we
> have to use an external timeout to decide that the sending has failed.
>
> Is there anyway that we can have sending bigger data (10MB) behaves the
> same as sending smaller data when the server suddenly shuts down: going to LWS_CALLBACK_CLOSED
> right away??
>
> Many thanks,
> Lan
>
>
>
> On Wed, Apr 1, 2015 at 8:50 AM, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:
>
>> Turn out that just removing #include stdint.h is enough and lws still
>> works fine.
>>
>> I made another change to allow using cert blobs instead of using cert
>> files, with cyassl. I can provide that as well?
>>
>> Should I email changes to you or push it to git?
>>
>> Thanks
>>
>> On Sat, Mar 28, 2015 at 5:34 AM, Andy Green <andy at warmcat.com> wrote:
>>
>>>
>>>
>>> On 28 March 2015 19:09:53 GMT+08:00, Duy Lan Nguyen <ndlan2k at gmail.com>
>>> wrote:
>>> >Hi Andy,
>>> >
>>> >Would it be possible to make this change?
>>>
>>> Sure, send a patch.
>>>
>>> The only windows this gets built on, an autobuilder, it builds already.
>>>
>>> If your patch makes it build for you and doesn't break the autobuilder
>>> then fine...
>>>
>>> -Andy
>>>
>>> >We'd greatly appreciate!!!
>>> >
>>> >On Thu, Mar 26, 2015 at 10:17 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>>> >wrote:
>>> >
>>> >> We need to use Visual Studio 2008 (VS 9) with libwebsockets to be
>>> >> compatible with our Driver DK. It was ok with previous lws but the
>>> >latest
>>> >> lws doesn't compile as libwebsockets.h includes stdint.h that's not
>>> >yet
>>> >> available in VS 9. Could you please fix that so that lws will work
>>> >with VS
>>> >> 9?
>>> >>
>>> >> Thanks!!
>>> >>
>>> >> On Wed, Mar 25, 2015 at 1:43 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>>> >wrote:
>>> >>
>>> >>> Awesome. Thanks
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Tue, Mar 24, 2015 at 10:55 AM, Andy Green <andy at warmcat.com>
>>> >wrote:
>>> >>>
>>> >>>>
>>> >>>>
>>> >>>> On 23 March 2015 22:28:45 GMT+08:00, Duy Lan Nguyen
>>> ><ndlan2k at gmail.com>
>>> >>>> wrote:
>>> >>>> >Yes, it's SSL connection
>>> >>>>
>>> >>>> I just pushed something that should solve it.
>>> >>>>
>>> >>>> -Andy
>>> >>>>
>>> >>>> >-----Original Message-----
>>> >>>> >From: "Andy Green" <andy at warmcat.com>
>>> >>>> >Sent: ‎3/‎23/‎2015 4:02 AM
>>> >>>> >To: "Duy Lan Nguyen" <ndlan2k at gmail.com>
>>> >>>> >Cc: "libwebsockets at ml.libwebsockets.org"
>>> >>>> ><libwebsockets at ml.libwebsockets.org>
>>> >>>> >Subject: Re: Atomic Frame Size
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >On 23 March 2015 15:57:30 GMT+08:00, Duy Lan Nguyen
>>> ><ndlan2k at gmail.com>
>>> >>>> >wrote:
>>> >>>> >>Hi Andy,
>>> >>>> >>
>>> >>>> >>I still have the problem of libwebsocket_context_destroy()
>>> >hanging,
>>> >>>> >but
>>> >>>> >>I
>>> >>>> >>still don't understand its cause. It happens as follows in my
>>> >Windows
>>> >>>> >>machine.
>>> >>>> >>
>>> >>>> >>In a client-server connection, the server is suddenly shut down.
>>> >The
>>> >>>> >>client
>>> >>>> >>doesn't realize it after a while, and finally closes down itself.
>>> >When
>>> >>>> >>the
>>> >>>> >>client calls libwebsocket_context_destroy(), it hangs in the
>>> >following
>>> >>>> >>loop
>>> >>>> >>inside libwebsocket_context_destroy()  as wsi is never NULL
>>> >>>> >>
>>> >>>> >>for (n = 0; n < context->fds_count; n++) {
>>> >>>> >>struct libwebsocket *wsi =
>>> >>>> >>context->lws_lookup[context->fds[n].fd];
>>> >>>> >>if (!wsi)
>>> >>>> >>continue;
>>> >>>> >>libwebsocket_close_and_free_session(context,
>>> >>>> >>wsi, LWS_CLOSE_STATUS_NOSTATUS /* no protocol close */);
>>> >>>> >>n--;
>>> >>>> >>}
>>> >>>> >>
>>> >>>> >>This happens as inside libwebsocket_close_and_free_session(), the
>>> >code
>>> >>>> >>keeps going into
>>> >>>> >>
>>> >>>> >>case WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE:
>>> >>>> >>if (wsi->truncated_send_len) {
>>> >>>> >>libwebsocket_callback_on_writable(context, wsi);
>>> >>>> >>return;
>>> >>>> >>}
>>> >>>> >>
>>> >>>> >>Did I do something wrong here or could you describe a solution to
>>> >this
>>> >>>> >
>>> >>>> >Probably nothing more wrong than using windows.
>>> >>>> >
>>> >>>> >>hanging problem?
>>> >>>> >
>>> >>>> >I guess there's a case where he has something he wants to send
>>> >before
>>> >>>> >dying, but he can't send it.  Is it an SSL connection?
>>> >>>> >
>>> >>>> >-Andy
>>> >>>> >
>>> >>>> >>Thanks,
>>> >>>> >>Lan
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>On Sun, Jan 25, 2015 at 8:13 PM, Andy Green <andy at warmcat.com>
>>> >wrote:
>>> >>>> >>
>>> >>>> >>>
>>> >>>> >>>
>>> >>>> >>> On 11 January 2015 17:15:56 GMT+08:00, Duy Lan Nguyen
>>> >>>> >><ndlan2k at gmail.com>
>>> >>>> >>> wrote:
>>> >>>> >>> >Hi Andy,
>>> >>>> >>> >
>>> >>>> >>> >I've implemented item 3). I can contribute it to lws if you
>>> >want.
>>> >>>> >>> >
>>> >>>> >>> >Do you have any news on items 1) and 2)?
>>> >>>> >>> >
>>> >>>> >>> >Thanks,
>>> >>>> >>> >Lan
>>> >>>> >>> >
>>> >>>> >>> >
>>> >>>> >>> >On Tue, Dec 23, 2014 at 7:26 PM, Duy Lan Nguyen
>>> ><ndlan2k at gmail.com>
>>> >>>> >>> >wrote:
>>> >>>> >>> >
>>> >>>> >>> >> In 1) and 2), I mean the server goes down suddenly
>>> >>>> >>> >>
>>> >>>> >>> >> On Tue, Dec 23, 2014 at 7:17 PM, Duy Lan Nguyen
>>> >>>> >><ndlan2k at gmail.com>
>>> >>>> >>> >wrote:
>>> >>>> >>> >>
>>> >>>> >>> >>> Hi Andy,
>>> >>>> >>> >>>
>>> >>>> >>> >>> I have some questions if you could help
>>> >>>> >>> >>> 1) I'm trying to let client handle gracefully when the
>>> >server
>>> >>>> >>shuts
>>> >>>> >>> >down.
>>> >>>> >>> >>> With lots of traffic going back and forth (each of client
>>> >and
>>> >>>> >>server
>>> >>>> >>> >has 2
>>> >>>> >>> >>> threads, one for sending and one for receiving),
>>> >>>> >>LWS_CALLBACK_CLOSED
>>> >>>> >>> >>> sometimes isn't caught by polling, and its callback routine
>>> >(in
>>> >>>> >>the
>>> >>>> >>> >>> switch-case) isn't called. Is it possible to make
>>> >>>> >>> >LWS_CALLBACK_CLOSED
>>> >>>> >>> >>> never be dropped?
>>> >>>> >>>
>>> >>>> >>> LWS never drops any close message AFAIK, but the OS may never
>>> >become
>>> >>>> >>aware
>>> >>>> >>> that the connection is dead.  It doesn't know if the reason
>>> >>>> >>everything is
>>> >>>> >>> quiet is because everything is quiet, or because somewhere a
>>> >network
>>> >>>> >>cable
>>> >>>> >>> fell out.
>>> >>>> >>>
>>> >>>> >>> You need to use keepalives of some sort (tcp, or PING / PONG in
>>> >lws)
>>> >>>> >>to
>>> >>>> >>> detect that.
>>> >>>> >>>
>>> >>>> >>> >>> 2) And after the connection is already shutdown for a while
>>> >>>> >>(client
>>> >>>> >>> >>> doesn't know it as LWS_CALLBACK_CLOSED is dropped), calling
>>> >>>> >>> >>> libwebsocket_context_destroy() is hanged. Is it possible to
>>> >>>> >avoid
>>> >>>> >>it
>>> >>>> >>> >>> hanging?
>>> >>>> >>>
>>> >>>> >>> Sorry where does it hang in there?
>>> >>>> >>>
>>> >>>> >>> >>> 3) We want to build a single-file client application, so
>>> >the app
>>> >>>> >>> >needs to
>>> >>>> >>> >>> take the server CA cert from a memory blob instead from a
>>> >file
>>> >>>> >>> >(info.
>>> >>>> >>> >>> ssl_ca_filepath). Can this be done?
>>> >>>> >>>
>>> >>>> >>> If you want to send a patch I'm happy to look at it.
>>> >>>> >>>
>>> >>>> >>> -Andy
>>> >>>> >>>
>>> >>>> >>> >>> Many thanks,
>>> >>>> >>> >>> Lan
>>> >>>> >>> >>>
>>> >>>> >>> >>>
>>> >>>> >>> >>>
>>> >>>> >>> >>> On Sat, Dec 13, 2014 at 10:50 AM, Andy Green
>>> ><andy at warmcat.com>
>>> >>>> >>> >wrote:
>>> >>>> >>> >>>
>>> >>>> >>> >>>>
>>> >>>> >>> >>>>
>>> >>>> >>> >>>> On 13 December 2014 23:42:18 GMT+08:00, Duy Lan Nguyen <
>>> >>>> >>> >>>> ndlan2k at gmail.com> wrote:
>>> >>>> >>> >>>> >Hi Andy,
>>> >>>> >>> >>>> >
>>> >>>> >>> >>>> >Just to make sure, the sample codes for the strategy you
>>> >>>> >>mentioned
>>> >>>> >>> >>>> >(wait
>>> >>>> >>> >>>> >for the WRITEABLE callback and the use a do { } while()
>>> >to
>>> >>>> >>stuff
>>> >>>> >>> >the
>>> >>>> >>> >>>> >pipe
>>> >>>> >>> >>>> >with a frame of total length below "the number" until
>>> >after
>>> >>>> >the
>>> >>>> >>> >last
>>> >>>> >>> >>>> >one,
>>> >>>> >>> >>>> >it has no more POLLOUT for you) are in test-server.c,
>>> >>>> >>> >>>> >in LWS_CALLBACK_HTTP_WRITEABLE and
>>> >>>> >>LWS_CALLBACK_SERVER_WRITEABLE,
>>> >>>> >>> >where
>>> >>>> >>> >>>> >you
>>> >>>> >>> >>>> >used
>>> >>>> >>> >>>> >
>>> >>>> >>> >>>> >if (lws_partial_buffered(wsi) ||
>>> >lws_send_pipe_choked(wsi)) {
>>> >>>> >>> >>>> >libwebsocket_callback_on_writable(context, wsi);
>>> >>>> >>> >>>> >break;
>>> >>>> >>> >>>> >}
>>> >>>> >>> >>>> >
>>> >>>> >>> >>>> >to check if "it has no more POLLOUT for you". Is that
>>> >right?
>>> >>>> >>> >>>>
>>> >>>> >>> >>>> Yes.  That way you can be sure you gave the kernel as much
>>> >as
>>> >>>> >it
>>> >>>> >>> >could
>>> >>>> >>> >>>> handle for that connection each time it had indicated you
>>> >could
>>> >>>> >>> >give the
>>> >>>> >>> >>>> connection anything.
>>> >>>> >>> >>>>
>>> >>>> >>> >>>> -Andy
>>> >>>> >>> >>>>
>>> >>>> >>> >>>> >Many thanks,
>>> >>>> >>> >>>> >Lan
>>> >>>> >>> >>>> >
>>> >>>> >>> >>>> >
>>> >>>> >>> >>>> >
>>> >>>> >>> >>>> >On Mon, Nov 24, 2014 at 2:45 PM, Andy Green
>>> ><andy at warmcat.com>
>>> >>>> >>> >wrote:
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> On 22 November 2014 16:42:32 GMT+08:00, Duy Lan Nguyen
>>> >>>> >>> >>>> ><ndlan2k at gmail.com>
>>> >>>> >>> >>>> >> wrote:
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> I am replying on the list.
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> >Hi Andy,
>>> >>>> >>> >>>> >> >
>>> >>>> >>> >>>> >> >I'll need to chop 1 MB message to atomic frames to
>>> >send and
>>> >>>> >>> >receive.
>>> >>>> >>> >>>> >It
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> What is an "atomic frame"... if you mean a logical
>>> >frame
>>> >>>> >that
>>> >>>> >>is
>>> >>>> >>> >>>> >split
>>> >>>> >>> >>>> >> into fragments of a larger logical message and sent
>>> >like
>>> >>>> >>that,
>>> >>>> >>> >yes
>>> >>>> >>> >>>> >you
>>> >>>> >>> >>>> >> should do that.
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> >seems that the default atomic frame size is 4KB which
>>> >is
>>> >>>> >>> >>>> >> >LWS_MAX_SOCKET_IO_BUF.
>>> >>>> >>> >>>> >> >
>>> >>>> >>> >>>> >> >Why should it be 4 KB? Or what should be the best
>>> >atomic
>>> >>>> >>frame
>>> >>>> >>> >size,
>>> >>>> >>> >>>> >> >for
>>> >>>> >>> >>>> >> >example for 1 MB message?
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> Well it depends on the server and the server load.
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> We avoid blocking by waiting for a POLLOUT indication.
>>> >But
>>> >>>> >>that
>>> >>>> >>> >only
>>> >>>> >>> >>>> >> tells us we could send "something", it doesn't say how
>>> >much
>>> >>>> >>we
>>> >>>> >>> >could
>>> >>>> >>> >>>> >send
>>> >>>> >>> >>>> >> without blocking.  If we get POLLOUT and try to send
>>> >10MB at
>>> >>>> >>> >once, we
>>> >>>> >>> >>>> >will
>>> >>>> >>> >>>> >> get a partial send.
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> The reason is the kernel is not willing to allocate
>>> >10MB
>>> >>>> >>buffer
>>> >>>> >>> >on
>>> >>>> >>> >>>> >kernel
>>> >>>> >>> >>>> >> side.  He did give us POLLOUT though, so he is willing
>>> >to
>>> >>>> >>take
>>> >>>> >>> >>>> >something.
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> Empirically at least with linux on typical ethernet and
>>> >WLAN
>>> >>>> >>> >setups,
>>> >>>> >>> >>>> >there
>>> >>>> >>> >>>> >> is "a number" below which we don't get a POLLOUT
>>> >indication.
>>> >>>> >
>>> >>>> >>It
>>> >>>> >>> >>>> >seems to
>>> >>>> >>> >>>> >> be around an ethernet frame size, eg, 1500 even if the
>>> >box
>>> >>>> >is
>>> >>>> >>> >heavily
>>> >>>> >>> >>>> >> loaded (at least, on my boxes).  If the box is less
>>> >loaded,
>>> >>>> >>it
>>> >>>> >>> >will
>>> >>>> >>> >>>> >accept
>>> >>>> >>> >>>> >> more.
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> So one strategy is shown in the test server code, wait
>>> >for
>>> >>>> >>the
>>> >>>> >>> >>>> >WRITEABLE
>>> >>>> >>> >>>> >> callback and the use a do { } while() to stuff the pipe
>>> >with
>>> >>>> >>a
>>> >>>> >>> >frame
>>> >>>> >>> >>>> >of
>>> >>>> >>> >>>> >> total length below "the number" until after the last
>>> >one, it
>>> >>>> >>has
>>> >>>> >>> >no
>>> >>>> >>> >>>> >more
>>> >>>> >>> >>>> >> POLLOUT for you.  Then if you still have more, ask for
>>> >>>> >>another
>>> >>>> >>> >>>> >WRITEABLE.
>>> >>>> >>> >>>> >> That way you should never block but also keep the
>>> >kernel
>>> >>>> >side
>>> >>>> >>as
>>> >>>> >>> >>>> >stuffed as
>>> >>>> >>> >>>> >> possible, and take advantage of packet coalesce if
>>> >possible.
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> Another strategy if you have a lot of memory is just
>>> >rely on
>>> >>>> >>the
>>> >>>> >>> >>>> >partial
>>> >>>> >>> >>>> >> send buffering in lws.  Do ask to send 1MB, when the
>>> >kernel
>>> >>>> >>just
>>> >>>> >>> >>>> >takes 2K
>>> >>>> >>> >>>> >> or whatever lws will copy the rest into a buffer and
>>> >take
>>> >>>> >>care
>>> >>>> >>> >of
>>> >>>> >>> >>>> >> prioritizing the send of the remaining buffered data
>>> >before
>>> >>>> >>that
>>> >>>> >>> >>>> >connection
>>> >>>> >>> >>>> >> will get another WRITABLE callback.  But 1) it's
>>> >expensive
>>> >>>> >>> >allocating
>>> >>>> >>> >>>> >and
>>> >>>> >>> >>>> >> copying around the buffer, and 2) when you use big
>>> >frames
>>> >>>> >you
>>> >>>> >>> >block
>>> >>>> >>> >>>> >the
>>> >>>> >>> >>>> >> link for control frames like PING for long periods or
>>> >from
>>> >>>> >>> >sending
>>> >>>> >>> >>>> >anything
>>> >>>> >>> >>>> >> else, so your latency can be hurt.
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> -Andy
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >> >Many thanks,
>>> >>>> >>> >>>> >> >Lan
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>> >>
>>> >>>> >>> >>>>
>>> >>>> >>> >>>>
>>> >>>> >>> >>>
>>> >>>> >>> >>
>>> >>>> >>>
>>> >>>> >>>
>>> >>>>
>>> >>>>
>>> >>>
>>> >>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20150411/2972ad0f/attachment-0001.html>


More information about the Libwebsockets mailing list