[Libwebsockets] Server stops reacting after some time

Andy Green andy at warmcat.com
Tue May 10 14:23:01 CEST 2016



On 05/10/2016 07:37 PM, Ofer Chesler wrote:
> Well,
> my data is an audio buffer with 1024 float numbers between 1 and -1.
> I did a small test with zip (on a sample data), and result is a little
> BIGGER than the original.

Zlib (it's the compresion library behind gzip) is not zip and if there 
is a quiet passage, there will be long regions with low entropy in the 
choice of sample data, ie, it will uncharacteristically be close to +/- 
0.0.  This is going to be more highly compressible.

> So when using compressions I didn't gain anything regarding size and lost a
> lot of performance.

Then don't use it.

However CPU load aside, it should be transparent if you are using it or not.

> The writeable callback was called very sparsely, and it these  occasions I
> tried to send back all the messages I had (like the mirror does in the test
> server) which probably caused the large message in the browser.

No, I think you missed the point.

A "message" is a logical entity in ws.

It's not like a packet where it may be coalesced, a message cannot 
change size in transit.

So the fact you end up with a 64KB 'ws message' in the browser means 
something else is wrong somewhere, if you think all you send are 4KB ws 
messages.

> Without the extensions the server ran for 90 minutes without any issue.

Well, good.

But to be clear there is only one extension, which is permessage-deflate.

If you don't want compression anyway you can just not use it, but I 
don't think we really understood your problem.

-Andy

> Ofer
>
> -----Original Message-----
> From: Andy Green [mailto:andy at warmcat.com]
> Sent: Tuesday, May 10, 2016 10:33 AM
> To: Ofer Chesler <ofer at whitenblue.com>; libwebsockets at ml.libwebsockets.org
> Subject: Re: [Libwebsockets] Server stops reacting after some time
>
>
>
> On 05/10/2016 03:02 PM, Ofer Chesler wrote:
>> Sorry for lack of the data,
>>
>> Here is more info: I'm working on windows 10, and the browser is
>> chrome, and I use version 1.7.3 I run the server in visual studio and
>> see no exception.
>> In the javascript console the only thing I see that I get back a very
>> large message (64k~), and then stuck. No other indications.
>
> Hm, well that's interesting.
>
> I guess there is a message size limit in Chrome, since he has to present the
> completed message all at once to the JS.
>
> How does your 4KB "block" relate to ws messages?  Or put another way, when
> you send from your lws user code, how do you manage message framing?
> Because it shouldn't be possible to accidentally accrete a 64KB ws message
> from 4KB messages ("message" has a very specific meaning for ws).  So I
> would look closely at how that is happening, instead of the subsequent
> stall.
>
>> I do see a lot of message "lws_rx_sm: doing draining flow" in the
>> server console.
>> I do see the negotiation on extensions and permessage-deflate selected.
>
> Yeah this is normal.  The "draining" that's going on is that zlib
> compression has inflated the incoming compressed data to something bigger
> than the compression output buffer (default 1KB).  In that case without
> using any more RX network data, there is more data that zlib wants to
> inflate to pass to the user code.  So he pauses taking network RX data and
> feeds your user code 1K or less buffers repeatedly out of zlib until it is
> drained.
>
> Consider, eg, you sent 4KB of 0x00, it only takes a few bytes across the
> network after zlib deflation, but at the receiver, they are inflated and
> then 4 x 1KB buffers full need to be 'drained': new network data must wait
> until we dealt with the glue of inflated data.
>
>> Now,
>> I did as you suggested and set info.extensions = NULL And from the
>> little I tested it seems to work (I get small messages back of 4k, and
>> no draining flow in console).
>> 1. Why is that ?
>
> You're the guy with the code ^^
>
>> 2. Currently I don't use SSL, but I will need to do. Can the issue
>> return when SSL enabled?
>
> Do you understand what "this issue" is?  I don't.
>
> 1.7.3 passes Autobahn torture-testing of permessage-deflate, if there were
> problems I think that would blow it up.
>
> I'm not suggesting turn off SSL because it's shaky but because with that and
> permessage-deflate off, you can more or less see the plaintext of the
> network traffic in tcpdump for your debugging activities.
>
> More or less, both permessage-deflate and SSL should be completely
> transparent for functionality and neither are shaky.
>
> -Andy
>
>> Ofer
>>
>> -----Original Message-----
>> From: Andy Green [mailto:andy at warmcat.com]
>> Sent: Tuesday, May 10, 2016 9:29 AM
>> To: Ofer Chesler <ofer at whitenblue.com>;
>> libwebsockets at ml.libwebsockets.org
>> Subject: Re: [Libwebsockets] Server stops reacting after some time
>>
>>
>>
>> On 05/10/2016 02:17 PM, Andy Green wrote:
>>>
>>>
>>> On 05/10/2016 02:06 PM, Ofer Chesler wrote:
>>>> Hello,
>>>>
>>>> I'm writing a websocket server which interacts with the browser.
>>>>
>>>> In the browser I garb audio (Web Audio API), sends it to the server,
>>>> which is doing some manipulations, and sends it back to the browser.
>>>>
>>>> In the browser I work with 512 samples each time and 2 channels,
>>>> which means I send each time around 512 (samples) * 4 (each one is
>>>> float) * 2
>>>> (channels) = 4096 bytes
>>>>
>>>> Assuming the audio have 44.1Khz sample rate, I need to send (and
>>>> receive) around  70 times a second.
>>>
>>> I see.
>>>
>>>> I wrote my server based on the test-server echo sample , and it
>>>> starts working, until at a certain point is suddenly stops. (the
>>>> browser tries to send more data but the server refuses to accept it).
>>>>
>>>> Any idea what might be the issue, or how can I debug it?
>>>
>>> Well... there is almost no information to be able to guess the problem.
>>>
>>> 1) What version of lws is this?  What platform?
>>>
>>> 2) I'd get the developer Javascript console up on the browser side
>>> and see what it thinks has happened.
>>>
>>> 3) On the server side, the first question would be 'has it crashed'?
>>> If it has crashed, maybe not with an abort, it can be interesting to
>>> run it under valgrind or gdb.
>>>
>>> 4) Enable cmake DEBUG mode cmake .. -DCMAKE_BUILD_TYPE=DEBUG and
>>> increase the logging using -d79 or so.
>>>
>>> 5) Find out what the server feels has happened... it feels the client
>>> connection closed?  Who closed it?  It's not closed?  The server
>>> doesn't see any more RX?  Or he can't do any more TX?
>>>
>>> 6) If permessage-deflate extension is negotiated, see if it's related
>>> to that by disabling it (info.extensions = NULL).
>>>
>>> 7) Likewise disable SSL if enabled
>>>
>>> 8) With no SSL or permessage-deflate, you can look for a clue by
>>> watching the network traffic directly with tcpdump (although browser
>>> ws payloads will be XORd, the header ws isn't) at the point it choked
>>
>> 9) Are you leaking something, like fd or sockets?  If on Linux have a
>> look in /proc/<PID>/fd to get a feel for what he has open.
>>
>> 10) Leaking memory?  On Linux watch it with top Shift-M to sort by
>> memory hogs and look at his process over time.
>>
>> 11) Remove complexity in the test by short-circuiting your audio
>> processing and just return him what he sent... problem is still coming?
>>
>> 12) Does the test server on the mirror test tab + run
>> libwebsockets-test-client -l localhost work well on your platform?  He
>> should spam crazy amounts of circles on the canvas and reconnect every
>> 10s or so.  This is also testing bulk data in both directions.
>>
>> 13) You should never try to send payload data outside the WRITABLE
> callback.
>>
>> -Andy
>>
>>> -Andy
>>>
>>>> Ofer
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Libwebsockets mailing list
>>>> Libwebsockets at ml.libwebsockets.org
>>>> http://libwebsockets.org/mailman/listinfo/libwebsockets
>>>>
>>> _______________________________________________
>>> Libwebsockets mailing list
>>> Libwebsockets at ml.libwebsockets.org
>>> http://libwebsockets.org/mailman/listinfo/libwebsockets
>>
>



More information about the Libwebsockets mailing list