[Libwebsockets] serving compressed http response from a zip archive

Andy Green andy at warmcat.com
Wed Feb 15 11:33:55 CET 2017

On 02/15/2017 05:59 PM, Denis Osvald wrote:
> On 2017-02-13 11:50, Andy Green wrote:
>> On 02/13/2017 06:31 PM, Denis Osvald wrote:
>>> Hello everybody,
>>> I'm using lws to make a web UI from a system with possibly constrained
>>> disk space. The resources (html, js, css) are stored on disk gzipped
>>> individually and content-encoding: gzip is added for such files when
>>> served by lws.
>>> Since this has (so far) been a special case not supported by lws,
>>> currently I'm handling this case "manually" by bypassing lws
>>> file-serving logic.
>>> I haven't looked at the recent changes in master, but do you think this
>>> use case (where GET /file.js ends up with /file.js.gz +
>>> content-encoding: gzip) would be resolved / made easier by new lws APIs?
>> Should be so.  Basically lws will signal to the VFS "open" in the flags
>> if gzip-coding is possible for the related client.
>> And the VFS "open" can signal back if it is providing something already
>> gzip-coded or not.
>> LWS file serving code will now deal with the headers both ways based on
>> what the client and the VFS open told in *flags.
>> So there is no bloat overall, since someone had to handle the headers
>> each way.  But now lws handles it in a standardized way.
> Okay, thanks for the info. I should take a look and try this new feature.
> Just another thought - the feature to support compressed serving is
> implemented through struct lws_plat_file_ops . Could it have been also
> implemented as an additional mount-protocol like LWSMPRO_(G)ZIP(PED)
> which would map a zipped resource directory into some path?

Well all the lws file serving stuff is using the vfs to touch the files 
already.  That's a good thing because lws now extends to devices like 
esp8266 where there is no posix and no filesystem by default.

I think no use for mount flag, because we either prepared gz versions of 
the files already or not.  If we did prepare them, and the client tells 
he can accept them, no reason not to serve them.

We could standardize everything about the gzip handling so there is no 
requirement to handle it in the vfs.

For example we can have a convention that if you have a file 'file', 
pre-gzipped files are always like 'file.gz'.  Then in the case both the 
peer can accept gz and there is a .gz version, lws can auto-prefer 
sending that with the correct headers.  You could even have both gz and 
plain versions stored if storage is cheaper than the decompression 
action on your system.

But IIRC Per was talking about fishing the files out of a libz blob, not 
having them as filesystem files.

That part of it is best done in the vfs layer I think and lws serving 
code does not have to know anything about it.  We can provide optional 
vfs fops that the user can wire up in his init code if he wants to store 
things like that.

But we could combine the .gz convention and the optional vfs 
manipulation if it isn't too burdensome to keep checking for possibly 
nonexistent .gz files.  In a generic server with a filesystem metadata 
cache I think it should be quite cheap.


> Of course, then you couldn't mix-and-match /a/b.js.gz and /a/c.js.
> To do this, you'd have to reinvent some kind of overlay and/or bind
> mounts inside lws anyway...
>>> If not (or if I have more special-case requirements), what is the best
>>> way to "bypass" lws's VFS / file-serving logic? Currently I'm doing it
>>> as a hack by making a mount of "/dev/null" which makes lws code fall
>>> through to my protocol handler, where I can handle special cases like
>>> this. It works alright this way.
>> The choice is clear in the v1 test server, either let lws do it or do it
>> manually like leaf.jpg.  In v2 it became more automatic with mounts or
>> do it manually.
>> It's great that hack worked but it's completely random that it did, it's
>> not intentional.
>> If there's something else useful to do in lws to support it we can talk
>> about if it makes sense.
> Okay. I'll see about that when I get around to porting the stuff to
> v2-style file serving.
>> -Andy
>>> Regards,
>>> Denis Osvald
>>> On 2017-02-12 11:54, Andy Green wrote:
>>>> On 02/12/2017 10:27 AM, Per Bothner wrote:
>>>>> On 02/11/2017 05:00 PM, Andy Green wrote:
>>>>>> On 12 February 2017 05:23:20 GMT+08:00, Per Bothner <per at bothner.com>
>>>>>> wrote:
>>>>>>> I'm using libwebsockets for my JavaScript-based DomTerm terminal
>>>>>>> emulator
>>>>>>> (http://domterm.org).  The "ldomterm" application is the
>>>>>>> combination of
>>>>>>> DomTerm + libwebsockets + application code (derived from ttyd), which
>>>>>>> optionally starts up a window in a specified browser.
>>>>>> Sounds very cool.
>>>>> Thanks - I'm pretty excited about it.
>>>>> I previously uses Java-WebSocket (or for qtdomterm I used Qt's internal
>>>>> WebSocket server). However, I'm happy to deprecate Java-WebSocket in
>>>>> favor of libwebsockets, for multiple reasons: The latter would
>>>>> typically
>>>>> use less resources; it is nice to have http and ws on the same port;
>>>>> and Java-WebSocket is not seeing any development.
>>>>> I'd like to give some credit to Shuanglei Tao, whose ttyd made it
>>>>> a lot easier to switch to libwebsockets.  ldomterm is basically ttyd
>>>>> but with xterm.js replaced by DomTerm.  (Plus I added options to
>>>>> automatically open a browser window.)
>>>>>> The main thing that will help stitch this in is lws file serving
>>>>>> arrangements have a thin VFS between the code doing the serving and
>>>>>> the filesystem, via a file_ops type struct of callbacks
>>>>>> https://github.com/warmcat/libwebsockets/blob/master/lib/libwebsockets.h#L4177
>>>>>> You can set your own file ops callback which provide normal posix
>>>>>> semantics to open / seek / read / close that hides the work you're
>>>>>> doing underneath.
>>>>> Thanks - I will look into serving from a zip archive - but initially
>>>>> without
>>>>> attempting to serve compressed data.  The latter will be a nice
>>>>> optimization
>>>>> that can be added when lws has the necessary support.
>>>> The suggested changes are added on master now.
>>>> I wasn't completely sure about the relationship between gzip and deflate
>>>> tokens in the headers.  What I did is added both "gzip" and "deflate"
>>>> need to be coming in to understand we could send gzip format, and if the
>>>> .open() handler chose to do it, we will send "gzip, deflate" as the
>>>> Content-Encoding.  If when you get to the point of testing it that's not
>>>> right, we can change it.
>>>> -Andy
>>>> -Andy
>>>>> I expect I'll have more questions soon.
>>>> _______________________________________________
>>>> Libwebsockets mailing list
>>>> Libwebsockets at ml.libwebsockets.org
>>>> https://libwebsockets.org/mailman/listinfo/libwebsockets
>>> _______________________________________________
>>> Libwebsockets mailing list
>>> Libwebsockets at ml.libwebsockets.org
>>> https://libwebsockets.org/mailman/listinfo/libwebsockets

More information about the Libwebsockets mailing list