[Libwebsockets] Truncated send handled by user not working as expected

nil100 at ig.com.br nil100 at ig.com.br
Mon Aug 18 15:35:03 CEST 2014


Em 18/08/2014 09:23, Andy Green escreveu: 

> On 18 August 2014 20:07:40 GMT+08:00, nil100 at ig.com.brwrote:
> Em 18/08/2014 08:25, Andy Green escreveu: On 18 August 2014 19:06:42 GMT+08:00, nil100 at ig.com.brwrote: Em 17/08/2014 21:56, Andy Green escreveu: On 18 August 2014 07:34:47 GMT+08:00, nil100 at ig.com.brwrote:Hello everybody, I'm using the library and in my case I want to handle truncated send myself but I ran into a situation. Linux's send function is returning -2 which in turn is causing libwebsockets to disregard my wish to handle truncated sends and consequently causes an assert failure later on. I'm not sure what your plan is for dealing with it in user code. I'm usually sending a large amount of data at a time and so I want to know when the pipe chokes and how many truncated sends it takes to completely send my whole chunk. Everything is happening in file output.c at line 130 which is the
 return from linux's native send (I'm not using SSL so
lws_ssl_capable_write is actually lws_ssl_capable_write_no_ssl). Now
when this returns LWS_SSL_CAPABLE_MORE_SERVICE (value -2) then the value
of variable "n" 

> So looking at the code, the actual send() didn't send anything just
 came back with -EAGAIN or similar. Then that function translates that
to a generic "I didn't send anything because I would have blocked" enum
return you mentioned. 

>> is set to 0 which causes the condition at line 172 (n &&
 wsi->u.ws.clean_buffer) to be false and continue execution as if 

> Yes I see. 
>> libwebsockets was handling truncates and consequent calls to fail
 assertion at line 112. 

> I don't see why it fails that assertion though, did you understand
 why during your debugging? Not having sent anything and buffer the
whole thing should have been okay. 

> So here the code enters the condition at line 107-110 because
 wsi->truncated_send_len && buf > (wsi->truncated_send_malloc +
wsi->truncated_send_len + wsi->truncated_send_offset). Now, the library
was not handling truncated sends in the first place (I configured it not
to)but it erroneously started handling after send returned an error I
dunno what this "configured it not to" thing is. I don't see anything
like that in the code (maybe I missed it). 

> and since I don't expect the library to do that I call
 libwebsocket_write again which makes the library believe I'm trying to
send new data when in fact I'm sending the rest of the data too. Yes --
that's exactly what's happening. So if you were OK with the lws
buffering (for sake of argument) you just need to open your sending loop
up to the service loop. Because the service loop is the only guy who can
regulate flushing the partial buffer without blocking. Actually if you
want to handle it, you have the same issue. If you just sit there
spamming you will also block, which makes your code only useful for one
connection or one socket fd at a time. 

> I'm a new libwebsockets user and wasn't sure how to proceed in
 getting this fixed. 

> There seems to be two separate issues here, one is you don't want it
 to buffer things if nothing got sent, and the other is you get an
assert firing when it does. 

> Correct. I don't want it to buffer things and the assert is firing
 because the library started handling things without me wanting it to. 

> If you remove the n && from line 172 it might do something towards
 what you want. But I am not sure if that leaves whatever blew the
assert just waiting to blow the assert another time. 

> What blew the assert was wsi->truncated_send_len not being 0 when it
 should have been. Sorry for the bad formatting. My webmail is crap.
Yeah it's pretty broken but it's fine as a text mode reply actually so
no worries. I think it's maybe a bit more complicated than you are
thinking. 1) What's actually sent on the wire may include protocol
prepending for websocket. That's tricky to account for but not
impossible. So the stuff at the send() is a different length than what
you sent in your user callback. 2) If there's an extension like
compression, which is commonly negotiated by browsers, what gets sent on
the wire - to the send() - is totally unrelated in size or content to
what you are sending from your user code. If you heard he managed to
send 1KByte of the compressed version, that's completely worthless
information to you because you don't know how big the remainder of the
compressed version is or what it was, from the user code with the
uncompressed buffer. These reasons are why we ended up with in-lws
transparent buffering, because especially the second one has no real
solution otherwise. -Andy 

> Cheers, Nilson N. da Silva
 Yes... I understand that there's more to sending over websockets than
just wiring to a socket and for that libwebsockets is very good (ofc
it's more than just websockets and thanks for creating it BTW). So what
I originally intended to do was something like the following: int m = 0;
size_t len = mybuffer_length; unsigned char *msg = mybuffer +
LWS_SEND_BUFFER_PRE_PADDING; do { int l = len - m; unsigned char *p =
msg + m; m += libwebsocket_write(wsi, p, l, LWS_WRITE_TEXT); /* do
something so that I know send was truncated and tell me how many
iterations it took to complete the whole chunk and how much was sent on
each */ } while(m < len); Now with the help Roger's comment if I have
"if (lws_send_pipe_choked(wsi)) continue;" inside this loop I get rid of
the assertion failure, but it seems libwebsockets is still handling
partials (I can see it the lib's logs). 

No it's still an approach that is incompatible with the lws buffering.

As soon as libwebsocket_write() figured it needed to buffer something,
you are not allowed to send anything new on libwebsocket_write() until
the buffered stuff is used up.

Instead you must return to the service loop, where lws will service its
buffer until it is empty, before giving you any new WRITEABLE callbacks.

And it's ONLY in your WRITEABLE callback you should do the
libwebsocket_write() activities.

It's generally enough in there if you see pipe_choked() is true, stop
sending right now, ask for a callback when you're writeable again and
leave the callback.

lws will do what needs to be done (using the service loop) and when it's
OK for you to send more, you will come back into the WRITEABLE callback
to send more.

If you adopt this approach to work with and through the service loop,
these problems will evaporate and the code will be super efficient and
low latency even with multiple connections blasting away.

There's some golden chunk size for your device, it's kernel, the memory
pressure state below which you won't normally buffer, say eg, 2KBytes.
So it's a lot cheaper to spam the connection until it would choke with
these small buffer writes that never show partial write problems, than
send 10MByte in one write which will definitely be a partial write and
need memcpys etc to do the buffering.

So please try adapt to this technique I think it will help a lot.


> Thank you guys for the discussion. Cheers, Nilson N. da Silva

Yes. I see... My intention was to find that golden chunk size and make
my code dynamically do some adaptations according to that value, which
was why I wanted to control partial writes myself. I will adopt your
technique and handle control back to lws when there's a choke. 

Thanks for all your input. 


Nilson N. da Silva 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20140818/783fcf27/attachment-0001.html>

More information about the Libwebsockets mailing list