[Libwebsockets] Server stops reacting after some time

Ofer Chesler ofer at whitenblue.com
Tue May 10 13:37:42 CEST 2016


Well,
my data is an audio buffer with 1024 float numbers between 1 and -1.
I did a small test with zip (on a sample data), and result is a little
BIGGER than the original.
So when using compressions I didn't gain anything regarding size and lost a
lot of performance.
The writeable callback was called very sparsely, and it these  occasions I
tried to send back all the messages I had (like the mirror does in the test
server) which probably caused the large message in the browser.

Without the extensions the server ran for 90 minutes without any issue.

Ofer

-----Original Message-----
From: Andy Green [mailto:andy at warmcat.com] 
Sent: Tuesday, May 10, 2016 10:33 AM
To: Ofer Chesler <ofer at whitenblue.com>; libwebsockets at ml.libwebsockets.org
Subject: Re: [Libwebsockets] Server stops reacting after some time



On 05/10/2016 03:02 PM, Ofer Chesler wrote:
> Sorry for lack of the data,
>
> Here is more info: I'm working on windows 10, and the browser is 
> chrome, and I use version 1.7.3 I run the server in visual studio and 
> see no exception.
> In the javascript console the only thing I see that I get back a very 
> large message (64k~), and then stuck. No other indications.

Hm, well that's interesting.

I guess there is a message size limit in Chrome, since he has to present the
completed message all at once to the JS.

How does your 4KB "block" relate to ws messages?  Or put another way, when
you send from your lws user code, how do you manage message framing?
Because it shouldn't be possible to accidentally accrete a 64KB ws message
from 4KB messages ("message" has a very specific meaning for ws).  So I
would look closely at how that is happening, instead of the subsequent
stall.

> I do see a lot of message "lws_rx_sm: doing draining flow" in the 
> server console.
> I do see the negotiation on extensions and permessage-deflate selected.

Yeah this is normal.  The "draining" that's going on is that zlib
compression has inflated the incoming compressed data to something bigger
than the compression output buffer (default 1KB).  In that case without
using any more RX network data, there is more data that zlib wants to
inflate to pass to the user code.  So he pauses taking network RX data and
feeds your user code 1K or less buffers repeatedly out of zlib until it is
drained.

Consider, eg, you sent 4KB of 0x00, it only takes a few bytes across the
network after zlib deflation, but at the receiver, they are inflated and
then 4 x 1KB buffers full need to be 'drained': new network data must wait
until we dealt with the glue of inflated data.

> Now,
> I did as you suggested and set info.extensions = NULL And from the 
> little I tested it seems to work (I get small messages back of 4k, and 
> no draining flow in console).
> 1. Why is that ?

You're the guy with the code ^^

> 2. Currently I don't use SSL, but I will need to do. Can the issue 
> return when SSL enabled?

Do you understand what "this issue" is?  I don't.

1.7.3 passes Autobahn torture-testing of permessage-deflate, if there were
problems I think that would blow it up.

I'm not suggesting turn off SSL because it's shaky but because with that and
permessage-deflate off, you can more or less see the plaintext of the
network traffic in tcpdump for your debugging activities.

More or less, both permessage-deflate and SSL should be completely
transparent for functionality and neither are shaky.

-Andy

> Ofer
>
> -----Original Message-----
> From: Andy Green [mailto:andy at warmcat.com]
> Sent: Tuesday, May 10, 2016 9:29 AM
> To: Ofer Chesler <ofer at whitenblue.com>; 
> libwebsockets at ml.libwebsockets.org
> Subject: Re: [Libwebsockets] Server stops reacting after some time
>
>
>
> On 05/10/2016 02:17 PM, Andy Green wrote:
>>
>>
>> On 05/10/2016 02:06 PM, Ofer Chesler wrote:
>>> Hello,
>>>
>>> I'm writing a websocket server which interacts with the browser.
>>>
>>> In the browser I garb audio (Web Audio API), sends it to the server, 
>>> which is doing some manipulations, and sends it back to the browser.
>>>
>>> In the browser I work with 512 samples each time and 2 channels, 
>>> which means I send each time around 512 (samples) * 4 (each one is
>>> float) * 2
>>> (channels) = 4096 bytes
>>>
>>> Assuming the audio have 44.1Khz sample rate, I need to send (and
>>> receive) around  70 times a second.
>>
>> I see.
>>
>>> I wrote my server based on the test-server echo sample , and it 
>>> starts working, until at a certain point is suddenly stops. (the 
>>> browser tries to send more data but the server refuses to accept it).
>>>
>>> Any idea what might be the issue, or how can I debug it?
>>
>> Well... there is almost no information to be able to guess the problem.
>>
>> 1) What version of lws is this?  What platform?
>>
>> 2) I'd get the developer Javascript console up on the browser side 
>> and see what it thinks has happened.
>>
>> 3) On the server side, the first question would be 'has it crashed'?
>> If it has crashed, maybe not with an abort, it can be interesting to 
>> run it under valgrind or gdb.
>>
>> 4) Enable cmake DEBUG mode cmake .. -DCMAKE_BUILD_TYPE=DEBUG and 
>> increase the logging using -d79 or so.
>>
>> 5) Find out what the server feels has happened... it feels the client 
>> connection closed?  Who closed it?  It's not closed?  The server 
>> doesn't see any more RX?  Or he can't do any more TX?
>>
>> 6) If permessage-deflate extension is negotiated, see if it's related 
>> to that by disabling it (info.extensions = NULL).
>>
>> 7) Likewise disable SSL if enabled
>>
>> 8) With no SSL or permessage-deflate, you can look for a clue by 
>> watching the network traffic directly with tcpdump (although browser 
>> ws payloads will be XORd, the header ws isn't) at the point it choked
>
> 9) Are you leaking something, like fd or sockets?  If on Linux have a 
> look in /proc/<PID>/fd to get a feel for what he has open.
>
> 10) Leaking memory?  On Linux watch it with top Shift-M to sort by 
> memory hogs and look at his process over time.
>
> 11) Remove complexity in the test by short-circuiting your audio 
> processing and just return him what he sent... problem is still coming?
>
> 12) Does the test server on the mirror test tab + run 
> libwebsockets-test-client -l localhost work well on your platform?  He 
> should spam crazy amounts of circles on the canvas and reconnect every 
> 10s or so.  This is also testing bulk data in both directions.
>
> 13) You should never try to send payload data outside the WRITABLE
callback.
>
> -Andy
>
>> -Andy
>>
>>> Ofer
>>>
>>>
>>>
>>> _______________________________________________
>>> Libwebsockets mailing list
>>> Libwebsockets at ml.libwebsockets.org
>>> http://libwebsockets.org/mailman/listinfo/libwebsockets
>>>
>> _______________________________________________
>> Libwebsockets mailing list
>> Libwebsockets at ml.libwebsockets.org
>> http://libwebsockets.org/mailman/listinfo/libwebsockets
>




More information about the Libwebsockets mailing list