[Libwebsockets] Server stops reacting after some time

Andy Green andy at warmcat.com
Tue May 10 09:33:07 CEST 2016



On 05/10/2016 03:02 PM, Ofer Chesler wrote:
> Sorry for lack of the data,
>
> Here is more info: I'm working on windows 10, and the browser is chrome, and
> I use version 1.7.3
> I run the server in visual studio and see no exception.
> In the javascript console the only thing I see that I get back a very large
> message (64k~), and then stuck. No other indications.

Hm, well that's interesting.

I guess there is a message size limit in Chrome, since he has to present 
the completed message all at once to the JS.

How does your 4KB "block" relate to ws messages?  Or put another way, 
when you send from your lws user code, how do you manage message 
framing?  Because it shouldn't be possible to accidentally accrete a 
64KB ws message from 4KB messages ("message" has a very specific meaning 
for ws).  So I would look closely at how that is happening, instead of 
the subsequent stall.

> I do see a lot of message "lws_rx_sm: doing draining flow" in the server
> console.
> I do see the negotiation on extensions and permessage-deflate selected.

Yeah this is normal.  The "draining" that's going on is that zlib 
compression has inflated the incoming compressed data to something 
bigger than the compression output buffer (default 1KB).  In that case 
without using any more RX network data, there is more data that zlib 
wants to inflate to pass to the user code.  So he pauses taking network 
RX data and feeds your user code 1K or less buffers repeatedly out of 
zlib until it is drained.

Consider, eg, you sent 4KB of 0x00, it only takes a few bytes across the 
network after zlib deflation, but at the receiver, they are inflated and 
then 4 x 1KB buffers full need to be 'drained': new network data must 
wait until we dealt with the glue of inflated data.

> Now,
> I did as you suggested and set info.extensions = NULL
> And from the little I tested it seems to work (I get small messages back of
> 4k, and no draining flow in console).
> 1. Why is that ?

You're the guy with the code ^^

> 2. Currently I don't use SSL, but I will need to do. Can the issue return
> when SSL enabled?

Do you understand what "this issue" is?  I don't.

1.7.3 passes Autobahn torture-testing of permessage-deflate, if there 
were problems I think that would blow it up.

I'm not suggesting turn off SSL because it's shaky but because with that 
and permessage-deflate off, you can more or less see the plaintext of 
the network traffic in tcpdump for your debugging activities.

More or less, both permessage-deflate and SSL should be completely 
transparent for functionality and neither are shaky.

-Andy

> Ofer
>
> -----Original Message-----
> From: Andy Green [mailto:andy at warmcat.com]
> Sent: Tuesday, May 10, 2016 9:29 AM
> To: Ofer Chesler <ofer at whitenblue.com>; libwebsockets at ml.libwebsockets.org
> Subject: Re: [Libwebsockets] Server stops reacting after some time
>
>
>
> On 05/10/2016 02:17 PM, Andy Green wrote:
>>
>>
>> On 05/10/2016 02:06 PM, Ofer Chesler wrote:
>>> Hello,
>>>
>>> I'm writing a websocket server which interacts with the browser.
>>>
>>> In the browser I garb audio (Web Audio API), sends it to the server,
>>> which is doing some manipulations, and sends it back to the browser.
>>>
>>> In the browser I work with 512 samples each time and 2 channels,
>>> which means I send each time around 512 (samples) * 4 (each one is
>>> float) * 2
>>> (channels) = 4096 bytes
>>>
>>> Assuming the audio have 44.1Khz sample rate, I need to send (and
>>> receive) around  70 times a second.
>>
>> I see.
>>
>>> I wrote my server based on the test-server echo sample , and it
>>> starts working, until at a certain point is suddenly stops. (the
>>> browser tries to send more data but the server refuses to accept it).
>>>
>>> Any idea what might be the issue, or how can I debug it?
>>
>> Well... there is almost no information to be able to guess the problem.
>>
>> 1) What version of lws is this?  What platform?
>>
>> 2) I'd get the developer Javascript console up on the browser side and
>> see what it thinks has happened.
>>
>> 3) On the server side, the first question would be 'has it crashed'?
>> If it has crashed, maybe not with an abort, it can be interesting to
>> run it under valgrind or gdb.
>>
>> 4) Enable cmake DEBUG mode cmake .. -DCMAKE_BUILD_TYPE=DEBUG and
>> increase the logging using -d79 or so.
>>
>> 5) Find out what the server feels has happened... it feels the client
>> connection closed?  Who closed it?  It's not closed?  The server
>> doesn't see any more RX?  Or he can't do any more TX?
>>
>> 6) If permessage-deflate extension is negotiated, see if it's related
>> to that by disabling it (info.extensions = NULL).
>>
>> 7) Likewise disable SSL if enabled
>>
>> 8) With no SSL or permessage-deflate, you can look for a clue by
>> watching the network traffic directly with tcpdump (although browser
>> ws payloads will be XORd, the header ws isn't) at the point it choked
>
> 9) Are you leaking something, like fd or sockets?  If on Linux have a look
> in /proc/<PID>/fd to get a feel for what he has open.
>
> 10) Leaking memory?  On Linux watch it with top Shift-M to sort by memory
> hogs and look at his process over time.
>
> 11) Remove complexity in the test by short-circuiting your audio processing
> and just return him what he sent... problem is still coming?
>
> 12) Does the test server on the mirror test tab + run
> libwebsockets-test-client -l localhost work well on your platform?  He
> should spam crazy amounts of circles on the canvas and reconnect every 10s
> or so.  This is also testing bulk data in both directions.
>
> 13) You should never try to send payload data outside the WRITABLE callback.
>
> -Andy
>
>> -Andy
>>
>>> Ofer
>>>
>>>
>>>
>>> _______________________________________________
>>> Libwebsockets mailing list
>>> Libwebsockets at ml.libwebsockets.org
>>> http://libwebsockets.org/mailman/listinfo/libwebsockets
>>>
>> _______________________________________________
>> Libwebsockets mailing list
>> Libwebsockets at ml.libwebsockets.org
>> http://libwebsockets.org/mailman/listinfo/libwebsockets
>



More information about the Libwebsockets mailing list