[Libwebsockets] Atomic Frame Size

Duy Lan Nguyen ndlan2k at gmail.com
Sat Apr 11 19:53:11 CEST 2015


Hi Andy,

We test how lws client handles server's sudden shutdown and observe the
following.

We use the method in call_back you advised before to keep the kernel
stuffed when sending (do while loop with lws_partial_buffered and
lws_send_pipe_choked).

    case LWS_CALLBACK_CLIENT_WRITEABLE:
do {
nTemp1 = (*ppSocketInfo)->m_SendInfo.m_cbSendData -
(*ppSocketInfo)->m_SendInfo.m_nBytesSent;

nTemp2 = (nTemp1 > ATOM_DATA_SIZE) ? ATOM_DATA_SIZE : nTemp1;

if
(memcpy_s(&(*ppSocketInfo)->m_SendInfo.m_rgbSendBuffer[LWS_SEND_BUFFER_PRE_PADDING],
ATOM_DATA_SIZE,
(*ppSocketInfo)->m_SendInfo.m_pbSendData +
(*ppSocketInfo)->m_SendInfo.m_nBytesSent,
nTemp2))
{
(*ppSocketInfo)->m_SendInfo.m_retSend = FAIL;
goto exit_send;
}

n = libwebsocket_write(wsi,
&(*ppSocketInfo)->m_SendInfo.m_rgbSendBuffer[LWS_SEND_BUFFER_PRE_PADDING],
nTemp2,
nOpts | LWS_WRITE_BINARY);

if (n < 0)
{
(*ppSocketInfo)->m_SendInfo.m_retSend = FAIL;
goto exit_send;
}
 (*ppSocketInfo)->m_SendInfo.m_nBytesSent += n;

if ((*ppSocketInfo)->m_SendInfo.m_nBytesSent >=
(*ppSocketInfo)->m_SendInfo.m_cbSendData)
{
goto exit_send;
}

}

if (lws_partial_buffered(wsi))
break;
} while (!lws_send_pipe_choked(wsi));

In case sending data is smaller (such as 100 KB), I guess this loop
completes faster. So when the server shuts down suddenly, the client side
immediately goes into case LWS_CALLBACK_CLOSED and stops right away.

In case sending data is big (such as 10 MB), the kernel seems never get
stuffed (the machine is pretty powerful), and the whole 10 MB just uses the
loop of a single case LWS_CALLBACK_CLIENT_WRITEABLE and it takes a bit
longer. So when the server shuts down suddenly, although the loop also
stops right away, the case LWS_CALLBACK_CLOSED is never reached. And we
have to use an external timeout to decide that the sending has failed.

Is there anyway that we can have sending bigger data (10MB) behaves the
same as sending smaller data when the server suddenly shuts down:
going to LWS_CALLBACK_CLOSED
right away??

Many thanks,
Lan



On Wed, Apr 1, 2015 at 8:50 AM, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:

> Turn out that just removing #include stdint.h is enough and lws still
> works fine.
>
> I made another change to allow using cert blobs instead of using cert
> files, with cyassl. I can provide that as well?
>
> Should I email changes to you or push it to git?
>
> Thanks
>
> On Sat, Mar 28, 2015 at 5:34 AM, Andy Green <andy at warmcat.com> wrote:
>
>>
>>
>> On 28 March 2015 19:09:53 GMT+08:00, Duy Lan Nguyen <ndlan2k at gmail.com>
>> wrote:
>> >Hi Andy,
>> >
>> >Would it be possible to make this change?
>>
>> Sure, send a patch.
>>
>> The only windows this gets built on, an autobuilder, it builds already.
>>
>> If your patch makes it build for you and doesn't break the autobuilder
>> then fine...
>>
>> -Andy
>>
>> >We'd greatly appreciate!!!
>> >
>> >On Thu, Mar 26, 2015 at 10:17 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>> >wrote:
>> >
>> >> We need to use Visual Studio 2008 (VS 9) with libwebsockets to be
>> >> compatible with our Driver DK. It was ok with previous lws but the
>> >latest
>> >> lws doesn't compile as libwebsockets.h includes stdint.h that's not
>> >yet
>> >> available in VS 9. Could you please fix that so that lws will work
>> >with VS
>> >> 9?
>> >>
>> >> Thanks!!
>> >>
>> >> On Wed, Mar 25, 2015 at 1:43 PM, Duy Lan Nguyen <ndlan2k at gmail.com>
>> >wrote:
>> >>
>> >>> Awesome. Thanks
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Mar 24, 2015 at 10:55 AM, Andy Green <andy at warmcat.com>
>> >wrote:
>> >>>
>> >>>>
>> >>>>
>> >>>> On 23 March 2015 22:28:45 GMT+08:00, Duy Lan Nguyen
>> ><ndlan2k at gmail.com>
>> >>>> wrote:
>> >>>> >Yes, it's SSL connection
>> >>>>
>> >>>> I just pushed something that should solve it.
>> >>>>
>> >>>> -Andy
>> >>>>
>> >>>> >-----Original Message-----
>> >>>> >From: "Andy Green" <andy at warmcat.com>
>> >>>> >Sent: ‎3/‎23/‎2015 4:02 AM
>> >>>> >To: "Duy Lan Nguyen" <ndlan2k at gmail.com>
>> >>>> >Cc: "libwebsockets at ml.libwebsockets.org"
>> >>>> ><libwebsockets at ml.libwebsockets.org>
>> >>>> >Subject: Re: Atomic Frame Size
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >On 23 March 2015 15:57:30 GMT+08:00, Duy Lan Nguyen
>> ><ndlan2k at gmail.com>
>> >>>> >wrote:
>> >>>> >>Hi Andy,
>> >>>> >>
>> >>>> >>I still have the problem of libwebsocket_context_destroy()
>> >hanging,
>> >>>> >but
>> >>>> >>I
>> >>>> >>still don't understand its cause. It happens as follows in my
>> >Windows
>> >>>> >>machine.
>> >>>> >>
>> >>>> >>In a client-server connection, the server is suddenly shut down.
>> >The
>> >>>> >>client
>> >>>> >>doesn't realize it after a while, and finally closes down itself.
>> >When
>> >>>> >>the
>> >>>> >>client calls libwebsocket_context_destroy(), it hangs in the
>> >following
>> >>>> >>loop
>> >>>> >>inside libwebsocket_context_destroy()  as wsi is never NULL
>> >>>> >>
>> >>>> >>for (n = 0; n < context->fds_count; n++) {
>> >>>> >>struct libwebsocket *wsi =
>> >>>> >>context->lws_lookup[context->fds[n].fd];
>> >>>> >>if (!wsi)
>> >>>> >>continue;
>> >>>> >>libwebsocket_close_and_free_session(context,
>> >>>> >>wsi, LWS_CLOSE_STATUS_NOSTATUS /* no protocol close */);
>> >>>> >>n--;
>> >>>> >>}
>> >>>> >>
>> >>>> >>This happens as inside libwebsocket_close_and_free_session(), the
>> >code
>> >>>> >>keeps going into
>> >>>> >>
>> >>>> >>case WSI_STATE_FLUSHING_STORED_SEND_BEFORE_CLOSE:
>> >>>> >>if (wsi->truncated_send_len) {
>> >>>> >>libwebsocket_callback_on_writable(context, wsi);
>> >>>> >>return;
>> >>>> >>}
>> >>>> >>
>> >>>> >>Did I do something wrong here or could you describe a solution to
>> >this
>> >>>> >
>> >>>> >Probably nothing more wrong than using windows.
>> >>>> >
>> >>>> >>hanging problem?
>> >>>> >
>> >>>> >I guess there's a case where he has something he wants to send
>> >before
>> >>>> >dying, but he can't send it.  Is it an SSL connection?
>> >>>> >
>> >>>> >-Andy
>> >>>> >
>> >>>> >>Thanks,
>> >>>> >>Lan
>> >>>> >>
>> >>>> >>
>> >>>> >>On Sun, Jan 25, 2015 at 8:13 PM, Andy Green <andy at warmcat.com>
>> >wrote:
>> >>>> >>
>> >>>> >>>
>> >>>> >>>
>> >>>> >>> On 11 January 2015 17:15:56 GMT+08:00, Duy Lan Nguyen
>> >>>> >><ndlan2k at gmail.com>
>> >>>> >>> wrote:
>> >>>> >>> >Hi Andy,
>> >>>> >>> >
>> >>>> >>> >I've implemented item 3). I can contribute it to lws if you
>> >want.
>> >>>> >>> >
>> >>>> >>> >Do you have any news on items 1) and 2)?
>> >>>> >>> >
>> >>>> >>> >Thanks,
>> >>>> >>> >Lan
>> >>>> >>> >
>> >>>> >>> >
>> >>>> >>> >On Tue, Dec 23, 2014 at 7:26 PM, Duy Lan Nguyen
>> ><ndlan2k at gmail.com>
>> >>>> >>> >wrote:
>> >>>> >>> >
>> >>>> >>> >> In 1) and 2), I mean the server goes down suddenly
>> >>>> >>> >>
>> >>>> >>> >> On Tue, Dec 23, 2014 at 7:17 PM, Duy Lan Nguyen
>> >>>> >><ndlan2k at gmail.com>
>> >>>> >>> >wrote:
>> >>>> >>> >>
>> >>>> >>> >>> Hi Andy,
>> >>>> >>> >>>
>> >>>> >>> >>> I have some questions if you could help
>> >>>> >>> >>> 1) I'm trying to let client handle gracefully when the
>> >server
>> >>>> >>shuts
>> >>>> >>> >down.
>> >>>> >>> >>> With lots of traffic going back and forth (each of client
>> >and
>> >>>> >>server
>> >>>> >>> >has 2
>> >>>> >>> >>> threads, one for sending and one for receiving),
>> >>>> >>LWS_CALLBACK_CLOSED
>> >>>> >>> >>> sometimes isn't caught by polling, and its callback routine
>> >(in
>> >>>> >>the
>> >>>> >>> >>> switch-case) isn't called. Is it possible to make
>> >>>> >>> >LWS_CALLBACK_CLOSED
>> >>>> >>> >>> never be dropped?
>> >>>> >>>
>> >>>> >>> LWS never drops any close message AFAIK, but the OS may never
>> >become
>> >>>> >>aware
>> >>>> >>> that the connection is dead.  It doesn't know if the reason
>> >>>> >>everything is
>> >>>> >>> quiet is because everything is quiet, or because somewhere a
>> >network
>> >>>> >>cable
>> >>>> >>> fell out.
>> >>>> >>>
>> >>>> >>> You need to use keepalives of some sort (tcp, or PING / PONG in
>> >lws)
>> >>>> >>to
>> >>>> >>> detect that.
>> >>>> >>>
>> >>>> >>> >>> 2) And after the connection is already shutdown for a while
>> >>>> >>(client
>> >>>> >>> >>> doesn't know it as LWS_CALLBACK_CLOSED is dropped), calling
>> >>>> >>> >>> libwebsocket_context_destroy() is hanged. Is it possible to
>> >>>> >avoid
>> >>>> >>it
>> >>>> >>> >>> hanging?
>> >>>> >>>
>> >>>> >>> Sorry where does it hang in there?
>> >>>> >>>
>> >>>> >>> >>> 3) We want to build a single-file client application, so
>> >the app
>> >>>> >>> >needs to
>> >>>> >>> >>> take the server CA cert from a memory blob instead from a
>> >file
>> >>>> >>> >(info.
>> >>>> >>> >>> ssl_ca_filepath). Can this be done?
>> >>>> >>>
>> >>>> >>> If you want to send a patch I'm happy to look at it.
>> >>>> >>>
>> >>>> >>> -Andy
>> >>>> >>>
>> >>>> >>> >>> Many thanks,
>> >>>> >>> >>> Lan
>> >>>> >>> >>>
>> >>>> >>> >>>
>> >>>> >>> >>>
>> >>>> >>> >>> On Sat, Dec 13, 2014 at 10:50 AM, Andy Green
>> ><andy at warmcat.com>
>> >>>> >>> >wrote:
>> >>>> >>> >>>
>> >>>> >>> >>>>
>> >>>> >>> >>>>
>> >>>> >>> >>>> On 13 December 2014 23:42:18 GMT+08:00, Duy Lan Nguyen <
>> >>>> >>> >>>> ndlan2k at gmail.com> wrote:
>> >>>> >>> >>>> >Hi Andy,
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >Just to make sure, the sample codes for the strategy you
>> >>>> >>mentioned
>> >>>> >>> >>>> >(wait
>> >>>> >>> >>>> >for the WRITEABLE callback and the use a do { } while()
>> >to
>> >>>> >>stuff
>> >>>> >>> >the
>> >>>> >>> >>>> >pipe
>> >>>> >>> >>>> >with a frame of total length below "the number" until
>> >after
>> >>>> >the
>> >>>> >>> >last
>> >>>> >>> >>>> >one,
>> >>>> >>> >>>> >it has no more POLLOUT for you) are in test-server.c,
>> >>>> >>> >>>> >in LWS_CALLBACK_HTTP_WRITEABLE and
>> >>>> >>LWS_CALLBACK_SERVER_WRITEABLE,
>> >>>> >>> >where
>> >>>> >>> >>>> >you
>> >>>> >>> >>>> >used
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >if (lws_partial_buffered(wsi) ||
>> >lws_send_pipe_choked(wsi)) {
>> >>>> >>> >>>> >libwebsocket_callback_on_writable(context, wsi);
>> >>>> >>> >>>> >break;
>> >>>> >>> >>>> >}
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >to check if "it has no more POLLOUT for you". Is that
>> >right?
>> >>>> >>> >>>>
>> >>>> >>> >>>> Yes.  That way you can be sure you gave the kernel as much
>> >as
>> >>>> >it
>> >>>> >>> >could
>> >>>> >>> >>>> handle for that connection each time it had indicated you
>> >could
>> >>>> >>> >give the
>> >>>> >>> >>>> connection anything.
>> >>>> >>> >>>>
>> >>>> >>> >>>> -Andy
>> >>>> >>> >>>>
>> >>>> >>> >>>> >Many thanks,
>> >>>> >>> >>>> >Lan
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >
>> >>>> >>> >>>> >On Mon, Nov 24, 2014 at 2:45 PM, Andy Green
>> ><andy at warmcat.com>
>> >>>> >>> >wrote:
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> On 22 November 2014 16:42:32 GMT+08:00, Duy Lan Nguyen
>> >>>> >>> >>>> ><ndlan2k at gmail.com>
>> >>>> >>> >>>> >> wrote:
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> I am replying on the list.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> >Hi Andy,
>> >>>> >>> >>>> >> >
>> >>>> >>> >>>> >> >I'll need to chop 1 MB message to atomic frames to
>> >send and
>> >>>> >>> >receive.
>> >>>> >>> >>>> >It
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> What is an "atomic frame"... if you mean a logical
>> >frame
>> >>>> >that
>> >>>> >>is
>> >>>> >>> >>>> >split
>> >>>> >>> >>>> >> into fragments of a larger logical message and sent
>> >like
>> >>>> >>that,
>> >>>> >>> >yes
>> >>>> >>> >>>> >you
>> >>>> >>> >>>> >> should do that.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> >seems that the default atomic frame size is 4KB which
>> >is
>> >>>> >>> >>>> >> >LWS_MAX_SOCKET_IO_BUF.
>> >>>> >>> >>>> >> >
>> >>>> >>> >>>> >> >Why should it be 4 KB? Or what should be the best
>> >atomic
>> >>>> >>frame
>> >>>> >>> >size,
>> >>>> >>> >>>> >> >for
>> >>>> >>> >>>> >> >example for 1 MB message?
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> Well it depends on the server and the server load.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> We avoid blocking by waiting for a POLLOUT indication.
>> >But
>> >>>> >>that
>> >>>> >>> >only
>> >>>> >>> >>>> >> tells us we could send "something", it doesn't say how
>> >much
>> >>>> >>we
>> >>>> >>> >could
>> >>>> >>> >>>> >send
>> >>>> >>> >>>> >> without blocking.  If we get POLLOUT and try to send
>> >10MB at
>> >>>> >>> >once, we
>> >>>> >>> >>>> >will
>> >>>> >>> >>>> >> get a partial send.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> The reason is the kernel is not willing to allocate
>> >10MB
>> >>>> >>buffer
>> >>>> >>> >on
>> >>>> >>> >>>> >kernel
>> >>>> >>> >>>> >> side.  He did give us POLLOUT though, so he is willing
>> >to
>> >>>> >>take
>> >>>> >>> >>>> >something.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> Empirically at least with linux on typical ethernet and
>> >WLAN
>> >>>> >>> >setups,
>> >>>> >>> >>>> >there
>> >>>> >>> >>>> >> is "a number" below which we don't get a POLLOUT
>> >indication.
>> >>>> >
>> >>>> >>It
>> >>>> >>> >>>> >seems to
>> >>>> >>> >>>> >> be around an ethernet frame size, eg, 1500 even if the
>> >box
>> >>>> >is
>> >>>> >>> >heavily
>> >>>> >>> >>>> >> loaded (at least, on my boxes).  If the box is less
>> >loaded,
>> >>>> >>it
>> >>>> >>> >will
>> >>>> >>> >>>> >accept
>> >>>> >>> >>>> >> more.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> So one strategy is shown in the test server code, wait
>> >for
>> >>>> >>the
>> >>>> >>> >>>> >WRITEABLE
>> >>>> >>> >>>> >> callback and the use a do { } while() to stuff the pipe
>> >with
>> >>>> >>a
>> >>>> >>> >frame
>> >>>> >>> >>>> >of
>> >>>> >>> >>>> >> total length below "the number" until after the last
>> >one, it
>> >>>> >>has
>> >>>> >>> >no
>> >>>> >>> >>>> >more
>> >>>> >>> >>>> >> POLLOUT for you.  Then if you still have more, ask for
>> >>>> >>another
>> >>>> >>> >>>> >WRITEABLE.
>> >>>> >>> >>>> >> That way you should never block but also keep the
>> >kernel
>> >>>> >side
>> >>>> >>as
>> >>>> >>> >>>> >stuffed as
>> >>>> >>> >>>> >> possible, and take advantage of packet coalesce if
>> >possible.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> Another strategy if you have a lot of memory is just
>> >rely on
>> >>>> >>the
>> >>>> >>> >>>> >partial
>> >>>> >>> >>>> >> send buffering in lws.  Do ask to send 1MB, when the
>> >kernel
>> >>>> >>just
>> >>>> >>> >>>> >takes 2K
>> >>>> >>> >>>> >> or whatever lws will copy the rest into a buffer and
>> >take
>> >>>> >>care
>> >>>> >>> >of
>> >>>> >>> >>>> >> prioritizing the send of the remaining buffered data
>> >before
>> >>>> >>that
>> >>>> >>> >>>> >connection
>> >>>> >>> >>>> >> will get another WRITABLE callback.  But 1) it's
>> >expensive
>> >>>> >>> >allocating
>> >>>> >>> >>>> >and
>> >>>> >>> >>>> >> copying around the buffer, and 2) when you use big
>> >frames
>> >>>> >you
>> >>>> >>> >block
>> >>>> >>> >>>> >the
>> >>>> >>> >>>> >> link for control frames like PING for long periods or
>> >from
>> >>>> >>> >sending
>> >>>> >>> >>>> >anything
>> >>>> >>> >>>> >> else, so your latency can be hurt.
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> -Andy
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >> >Many thanks,
>> >>>> >>> >>>> >> >Lan
>> >>>> >>> >>>> >>
>> >>>> >>> >>>> >>
>> >>>> >>> >>>>
>> >>>> >>> >>>>
>> >>>> >>> >>>
>> >>>> >>> >>
>> >>>> >>>
>> >>>> >>>
>> >>>>
>> >>>>
>> >>>
>> >>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20150411/9cece0ec/attachment-0001.html>


More information about the Libwebsockets mailing list