[Libwebsockets] Atomic Frame Size

Andy Green andy at warmcat.com
Thu Nov 27 01:13:20 CET 2014

On 27 November 2014 07:57:49 GMT+08:00, Duy Lan Nguyen <ndlan2k at gmail.com> wrote:
>It seems that  libwebsocket_write(wsi,
>&pbSendData[LWS_SEND_BUFFER_PRE_PADDING], cbSendData, s_nOpts |
>also changes the values in buffer pbSendData.
>Can we do it without changing pbSendData?
>The benefit is that I can put data directly on pbSendData, and if I
>receive an ACK (I implement) I can just resend the buffer pbSendData.

I understand what you're asking and it's reasonable.

But websockets specifies that transmit data should be "masked", it means XORed with a random 32-bit key per frame.

We do the masking in-place so it doesn't take up any extra memory.

Because ws is on TCP/IP with reliable delivery, normally there's no worry about resend.

If that's something to worry about for you, as it is you need to transmit a copy of the data.

We could make an option where the masked data gets written to a temp buffer before the actual write... it is more expensive but not tremendously so.  But a lot of confusion will be coming about how to size that temp buffer since there's a big variety of approaches in the user code to sending stuff.


>Many thanks,
>On Mon, Nov 24, 2014 at 2:45 PM, Andy Green <andy at warmcat.com> wrote:
>> On 22 November 2014 16:42:32 GMT+08:00, Duy Lan Nguyen
><ndlan2k at gmail.com>
>> wrote:
>> I am replying on the list.
>> >Hi Andy,
>> >
>> >I'll need to chop 1 MB message to atomic frames to send and receive.
>> What is an "atomic frame"... if you mean a logical frame that is
>> into fragments of a larger logical message and sent like that, yes
>> should do that.
>> >seems that the default atomic frame size is 4KB which is
>> >
>> >Why should it be 4 KB? Or what should be the best atomic frame size,
>> >for
>> >example for 1 MB message?
>> Well it depends on the server and the server load.
>> We avoid blocking by waiting for a POLLOUT indication.  But that only
>> tells us we could send "something", it doesn't say how much we could
>> without blocking.  If we get POLLOUT and try to send 10MB at once, we
>> get a partial send.
>> The reason is the kernel is not willing to allocate 10MB buffer on
>> side.  He did give us POLLOUT though, so he is willing to take
>> Empirically at least with linux on typical ethernet and WLAN setups,
>> is "a number" below which we don't get a POLLOUT indication.  It
>seems to
>> be around an ethernet frame size, eg, 1500 even if the box is heavily
>> loaded (at least, on my boxes).  If the box is less loaded, it will
>> more.
>> So one strategy is shown in the test server code, wait for the
>> callback and the use a do { } while() to stuff the pipe with a frame
>> total length below "the number" until after the last one, it has no
>> POLLOUT for you.  Then if you still have more, ask for another
>> That way you should never block but also keep the kernel side as
>stuffed as
>> possible, and take advantage of packet coalesce if possible.
>> Another strategy if you have a lot of memory is just rely on the
>> send buffering in lws.  Do ask to send 1MB, when the kernel just
>takes 2K
>> or whatever lws will copy the rest into a buffer and take care of
>> prioritizing the send of the remaining buffered data before that
>> will get another WRITABLE callback.  But 1) it's expensive allocating
>> copying around the buffer, and 2) when you use big frames you block
>> link for control frames like PING for long periods or from sending
>> else, so your latency can be hurt.
>> -Andy
>> >Many thanks,
>> >Lan

More information about the Libwebsockets mailing list