[Libwebsockets] permessage-deflate

Andy Green andy at warmcat.com
Mon Jan 11 07:00:46 CET 2016

On January 4, 2016 11:56:13 PM GMT+00:00, Andy Green <andy at warmcat.com> wrote:
>Hi -
>The remaining tests in Autobahn rely on permessage-deflate, which was 
>standardized a few weeks ago as RFC7692
>It's basically the same as the existing deflate-frame extension we
>but clearly the old extensions are dead (deflate-stream has been 
>disabled from build for more than a year) or dying.
>However the implementation of deflate-frame that we have takes the 
>approach to insist to deflate and inflate whole messages at once, using
>malloc / realloc until it will fit.
>Autobahn sends large messages in fragments and expects them to be 
>streamed back... it's much more useful if the compression extension can
>handle the input and output as it comes without large buffers or limit 
>on message size.
>I have implemented the changes needed under the hood (only under he 
>hood...) so that TX path extensions can chunk their output through the 
>event loop.  And I implemented permessage-deflate here with RX chunking
>It can work with Chrome and the test server + client, but it still
>about half the compression-related Autobahn tests.  Much of that is 
>because I didn't implement the option parsing in RFC7692 yet, but some 
>of them are corner cases in the chunking that still need fixing. 
>Unfortunately these remaining problems occur depending on compression 
>fragment size after tens of MB of logs from what Autobahn's sending.
>So my current plan is not push this stuff at the moment, and make a
>jig I can control better than Autobahn to be able to provoke all 
>possible chunking / framing conditions at will, rather than when the 
>data Autobahn sends randomly generates it.  After it hopefully passes 
>that I will go back to Autobahn.  This stuff was very difficult 
>originally, that is how come it ended up with the full message copout 
>implementation, but this new implementation is much cleaner.
>Unfortunately I am out of free time to work on this, I'll resume next 

This is now done, except for 2.10 (ping spamming that is not in the standard) we pass *everything* in Autobahn ws fuzzing server including ALL the permessage-deflate variations.


And we do it using what is now the default pmce chunking of 1024 in each direction, that is the user gets incoming inflated data as soon as 1024 bytes or less is ready, and since the chunking goes through the event loop, he can send things that will be sent as soon as 1024 bytes or less of deflated data is ready, interleaved with rx chunk reception.  So the memory overhead of processing large compressed messages is gone, you never need more than one chunk at a time allocated.

The test server now shows in the html which extensions were negotiated; this works in chrome + ffox with test server and client using permessage-deflate now.

Extension public apis are now always available, to allow code that knows about the extension to work without it faced with a library compiled without extension support.

User code is responsible for defining which extensions are active and what options are offered.  The api to get internal extensions that was used before still exists, but returns NULL meaning extensions are disabled until the code is updated.  The test client + server and echo show how to do.  These changes are mandated for extension option support required in rfc7692 and required by Autobahn.  Basically the client user code declares in a string which options it prefers and the server's response is parsed and passed to the extension callback.

Extensions are now responsible to allocate and free their own private instance allocation meaning that the related struct definition can stay away from the public api.

I added a new client connect api that uses an info type struct instead of a bazillion parameters.  The two existing apis now map on to the new one internally so there is no change, except they are marked deprecated.  Extension option support mandates the client initiating the extension option offer negotiation.

This is quite a big patch

 40 files changed, 1914 insertions(+), 1133 deletions(-)

...but it only really affects extensions, which were effectively broken for a while by deflate-frame being replaced out in the real world by permessage-deflate.

And at the end of it we now have a compliant permessage-deflate which passes Autobahn (and travis + appveyor).

If you have been wanting to use link compression, please test it out.


>Libwebsockets mailing list
>Libwebsockets at ml.libwebsockets.org

More information about the Libwebsockets mailing list