[Libwebsockets] Truncated send handled by user not working as expected

Andy Green andy at warmcat.com
Mon Aug 18 14:23:18 CEST 2014

On 18 August 2014 20:07:40 GMT+08:00, nil100 at ig.com.br wrote:
>Em 18/08/2014 08:25, Andy Green escreveu: 
>> On 18 August 2014 19:06:42 GMT+08:00, nil100 at ig.com.brwrote:
>> Em 17/08/2014 21:56, Andy Green escreveu: On 18 August 2014 07:34:47
>GMT+08:00, nil100 at ig.com.brwrote: Hello everybody, I'm using the
>library and in my case I want to
> handle truncated send myself but I ran into a situation. Linux's send
>function is returning -2 which in turn is causing libwebsockets to
>disregard my wish to handle truncated sends and consequently causes an
>assert failure later on. 
>> I'm not sure what your plan is for dealing with it in user code. I'm
>usually sending a large amount of data at a time and so I want to
> know when the pipe chokes and how many truncated sends it takes to
>completely send my whole chunk. 
>>> Everything is happening in file output.c at line 130 which is the
> return from linux's native send (I'm not using SSL so
>lws_ssl_capable_write is actually lws_ssl_capable_write_no_ssl). Now
>when this returns LWS_SSL_CAPABLE_MORE_SERVICE (value -2) then the
>of variable "n" 
>> So looking at the code, the actual send() didn't send anything just
> came back with -EAGAIN or similar. Then that function translates that
>to a generic "I didn't send anything because I would have blocked" enum
>return you mentioned. 
>>> is set to 0 which causes the condition at line 172 (n &&
> wsi->u.ws.clean_buffer) to be false and continue execution as if 
>> Yes I see. 
>>> libwebsockets was handling truncates and consequent calls to fail
> assertion at line 112. 
>> I don't see why it fails that assertion though, did you understand
> why during your debugging? Not having sent anything and buffer the
>whole thing should have been okay. 
>> So here the code enters the condition at line 107-110 because
> wsi->truncated_send_len && buf > (wsi->truncated_send_malloc +
>wsi->truncated_send_len + wsi->truncated_send_offset). Now, the library
>was not handling truncated sends in the first place (I configured it
>to)but it erroneously started handling after send returned an error 
>I dunno what this "configured it not to" thing is. I don't see anything
>like that in the code (maybe I missed it).
>> and since I don't expect the library to do that I call
>libwebsocket_write again which makes the library believe I'm trying to
>send new data when in fact I'm sending the rest of the data too.
>Yes -- that's exactly what's happening.
>So if you were OK with the lws buffering (for sake of argument) you
>need to open your sending loop up to the service loop.
>Because the service loop is the only guy who can regulate flushing the
>partial buffer without blocking.
>Actually if you want to handle it, you have the same issue. If you just
>sit there spamming you will also block, which makes your code only
>useful for one connection or one socket fd at a time.
>> I'm a new libwebsockets user and wasn't sure how to proceed in
> getting this fixed. 
>> There seems to be two separate issues here, one is you don't want it
> to buffer things if nothing got sent, and the other is you get an
>assert firing when it does. 
>> Correct. I don't want it to buffer things and the assert is firing
> because the library started handling things without me wanting it to. 
>> If you remove the n && from line 172 it might do something towards
> what you want. But I am not sure if that leaves whatever blew the
>assert just waiting to blow the assert another time. 
>> What blew the assert was wsi->truncated_send_len not being 0 when it
> should have been. Sorry for the bad formatting. My webmail is crap. 
>Yeah it's pretty broken but it's fine as a text mode reply actually so
>no worries.
>I think it's maybe a bit more complicated than you are thinking.
>1) What's actually sent on the wire may include protocol prepending for
>websocket. That's tricky to account for but not impossible. So the
>at the send() is a different length than what you sent in your user
>2) If there's an extension like compression, which is commonly
>negotiated by browsers, what gets sent on the wire - to the send() - is
>totally unrelated in size or content to what you are sending from your
>user code.
>If you heard he managed to send 1KByte of the compressed version,
>completely worthless information to you because you don't know how big
>the remainder of the compressed version is or what it was, from the
>code with the uncompressed buffer.
>These reasons are why we ended up with in-lws transparent buffering,
>because especially the second one has no real solution otherwise.
>> Cheers, Nilson N. da Silva
>Yes... I understand that there's more to sending over websockets than
>just wiring to a socket and for that libwebsockets is very good (ofc
>it's more than just websockets and thanks for creating it BTW). 
>So what I originally intended to do was something like the following: 
>int m = 0;
> size_t len = mybuffer_length;
> unsigned char *msg = mybuffer + LWS_SEND_BUFFER_PRE_PADDING;
> do {
> int l = len - m;
> unsigned char *p = msg + m;
> m += libwebsocket_write(wsi, p, l, LWS_WRITE_TEXT);
> /*
> do something so that I know send was truncated and tell me
> how many iterations it took to complete the whole chunk
> and how much was sent on each
> */
>} while(m < len); 
>Now with the help Roger's comment if I have "if
>(lws_send_pipe_choked(wsi)) continue;" inside this loop I get rid of
>assertion failure, but it seems libwebsockets is still handling
>(I can see it the lib's logs). 

No it's still an approach that is incompatible with the lws buffering.

As soon as libwebsocket_write() figured it needed to buffer something, you are not allowed to send anything new on libwebsocket_write() until the buffered stuff is used up.

Instead you must return to the service loop, where lws will service its buffer until it is empty, before giving you any new WRITEABLE callbacks.

And it's ONLY in your WRITEABLE callback you should do the libwebsocket_write() activities.

It's generally enough in there if you see pipe_choked() is true, stop sending right now, ask for a callback when you're writeable again and leave the callback.

lws will do what needs to be done (using the service loop) and when it's OK for you to send more, you will come back into the WRITEABLE callback to send more.

If you adopt this approach to work with and through the service loop, these problems will evaporate and the code will be super efficient and low latency even with multiple connections blasting away.

There's some golden chunk size for your device, it's kernel, the memory pressure state below which you won't normally buffer, say eg, 2KBytes.  So it's a lot cheaper to spam the connection until it would choke with these small buffer writes that never show partial write problems, than send 10MByte in one write which will definitely be a partial write and need memcpys etc to do the buffering.

So please try adapt to this technique I think it will help a lot.


>Thank you guys for the discussion. 
>Nilson N. da Silva 

More information about the Libwebsockets mailing list