[Libwebsockets] Packet size question

Andy Green andy at warmcat.com
Wed May 20 19:59:59 CEST 2020



On 5/20/20 6:21 PM, Brice Hamon wrote:
> Hi Andy,
> 
> We've implemented the message fragmentation to regular MTU size 
> - LWS_PRE so 1484.
> 
> We chopped the big messages in chunks of 1484 and send it with proto 
> = LWS_WRITE_BINARY | LWS_WRITE_NO_FIN.
> Then the last one with only proto = LWS_WRITE_BINARY.
> What happens next is that we are getting a LWS_CALLBACK_CLOSED and the 
> connection gets closed, which is not good for our application as we are 
> not over http.

That's not how ws works, it needs a BINARY on the first and CONTINUATION 
on the subsequent ones.

There is a helper api here

https://libwebsockets.org/git/libwebsockets/tree/include/libwebsockets/lws-write.h#n221-246

that makes it easier to form the right flags.

-Andy

> If we send all packets with only LWS_WRITE_BINARY, all works well, and 
> we can send huge buffers across.
> 
> So I don't know if our implementation is correct but so far sending the 
> big message in small chunks allowed us to send big messages,
> we just don't use the LWS_WRITE_NO_FIN flag.
> 
> Thank you
> Brice.
> 
> 
> 
> 
> 
> 
> 
> 
> On Wed, May 20, 2020 at 8:33 AM Brice Hamon <normandviking at gmail.com 
> <mailto:normandviking at gmail.com>> wrote:
> 
>     Thanks Andy.
>     We will go the safer route and ask for write readiness before
>     actually calling lws_write().
>     Thanks again,
>     Brice.
> 
> 
>     On Wed, May 20, 2020 at 8:22 AM Andy Green <andy at warmcat.com
>     <mailto:andy at warmcat.com>> wrote:
> 
> 
> 
>         On 5/20/20 1:13 PM, Brice Hamon wrote:
>          > Before we make modifications, what is the best approach?
>          > To chop the message in MTU chunks and call lws_write() for
>         each packet
>          > after we get a WRITE notification, or call lws_write() in a
>         loop until
>          > we get an error like TCP, then go back to the epoll loop for
>         the remaining?
> 
>         If at all possible, do not create / read / produce / store the
>         output
>         until it's able to send it, and then only the part that you are
>         going to
>         send.
> 
>         The best way to deal with it is ask for a new writeable callback
>         for
>         each chunk.  That will adapt to all kinds of situations cleanly
>         like
>         many connections sending, sending as h2 streams, ws-over-h2 etc.
> 
>         On a particular machine under its normal conditions you might
>         find it's
>         no problem to send larger chunks up to several KB.  If it does
>         occasionally meet a partial send, lws will conceal it (by
>         stopping you
>         seeing any more writeable callbacks until it has cleared it in the
>         background).  You want some sweet spot between throughput and
>         probability of having the partials using heap, whatever that is
>         for your
>         particular situation.
> 
>         -Andy
> 
>          > Thank you.
>          > Brice.
>          >
>          > On Tue, May 19, 2020 at 5:53 PM Brice Hamon
>         <normandviking at gmail.com <mailto:normandviking at gmail.com>
>          > <mailto:normandviking at gmail.com
>         <mailto:normandviking at gmail.com>>> wrote:
>          >
>          >     Hi Andy,
>          >
>          >     Ok got it. Thank you.
>          >
>          >     Yes it is withing ws.
>          >     So I'll chop the big buffers into MTU size packets and
>         lws_write()
>          >     them, first n-1 with the FIN flag, and the last one without.
>          >     I'll report.
>          >     Thank you
>          >     Brice.
>          >
>          >
>          >
>          >
>          >     On Tue, May 19, 2020 at 5:33 PM Andy Green
>         <andy at warmcat.com <mailto:andy at warmcat.com>
>          >     <mailto:andy at warmcat.com <mailto:andy at warmcat.com>>> wrote:
>          >
>          >
>          >
>          >         On 5/19/20 9:52 PM, Brice Hamon wrote:
>          >          > Hi guys,
>          >          >
>          >          > I was wondering about payload size in lws.
>          >          >
>          >          > Is there a maximum size pre-defined, or I can
>         allocate a big
>          >         buffer and
>          >          > use lws_write() to let lws manage the fragmentation?
>          >          >
>          >          > So far we are hitting around a 273872 chars limit
>         somewhere.
>          >
>          >         No, should should definitely chop it up yourself into
>         something
>          >         reasonable.  lws doesn't have any fixed limit but if your
>          >         connection
>          >         gets choked, space will appear in the tcp window as
>         things are
>          >         acked, if
>          >         the connection is relatively slow that will manifest
>         itself as
>          >         the tcp
>          >         stack being willing to send "a bit more" measured in
>         mtu-sized
>          >         packets.
>          >         Exactly how much it will accept depends on the
>         platform, memory
>          >         pressure
>          >         and the state of the tcp window, but typically
>         somewhere between
>          >         1 and 2
>          >         mtu is a good bet, so chunks of 1.4 .. 2.8KB.
>          >
>          >         If you try to send giant buffers it all goes on the
>         heap and
>          >         it's very
>          >         inefficient in that for every send(), the whole of
>         the giant
>          >         buffer is
>          >         copied into kernel space each time, even if it ends
>         up the
>          >         network stack
>          >         only used 2KB of it.
>          >
>          >         If it's ws protocol you can mark the writes after the
>         first as not
>          >         having a FIN, so it's a continuation of an existing
>         ws message
>          >         in its
>          >         own frame.
>          >
>          >         If it's eg http body you can just keep sending the
>         next part.
>          >
>          >         -Andy
>          >
>          >          > Thank you,
>          >          > Brice.
>          >          >
>          >          >
>          >          >
>          >          > _______________________________________________
>          >          > Libwebsockets mailing list
>          >          > Libwebsockets at ml.libwebsockets.org
>         <mailto:Libwebsockets at ml.libwebsockets.org>
>          >         <mailto:Libwebsockets at ml.libwebsockets.org
>         <mailto:Libwebsockets at ml.libwebsockets.org>>
>          >          >
>         https://libwebsockets.org/mailman/listinfo/libwebsockets
>          >          >
>          >
> 


More information about the Libwebsockets mailing list