[Libwebsockets] polling for file descriptor i/o

Per Bothner per at bothner.com
Mon Mar 6 21:01:52 CET 2017

On 03/06/2017 11:27 AM, Andy Green wrote:
>> However, it appears that every time lws_fops_zip_open is called, it
>> opens the
>> zipfile afresh, and *rescans* the central directory.
>> This seems very unfortunate.  In my implementation, the zipfile was
>> opened and its directory scanned just once.
> Yeah... but your implementation allocates an unbounded amount of memory then according to the directory size for an unbounded period (and allocates an unbounded amount of memory inflating the whole content into memory before serving it).

Nothing is inflated, unless the browser can't handle compressed files, and even in that
case it is only inflated temporarily.

> These characteristics mean that the resource-constrained case that can benefit most from this can't use it, especially platforms that cannot mmap the zip as it required.

I understand.  Of course I'm mostly focused on less resource-constrained systems:
The DomTerm JavaScript is pretty big, and depends on a modern browser, so assuming
mmap and plentiful memory is not much of a constraint.

> In lws implementation the total heap usage serving from inside a zip is restricted to a few hundred bytes including the decompress block buffer; the device can cheaply serve huge files from the zip bigger than its own memory, compressed or decompressed.  Even the weakest platforms can afford to use it.  And it uses platform fops only to access the zip file without needing mmap.

>> Of course one option is I could keep my mmap-based zip-reader
>> implementation and just
>> ignore the one in libwebsockets.  The amount of "code duplication"
>> would be trivial.
> It'd be a shame... once the zip has been read once, in Linux those pieces will be in memory in the block cache.  Seeking and reading from there is already doing it from ram without going back to the storage device.
> If some specific thing you need is objectively slow, we can discuss targeting that with a CMake option indicating on that platform, speed is more important than memory usage.  But I suggest measuring it and identifying what is too slow first so we discuss the right thing.

Ok.  You're probably right.

It might be useful in some environments to statically link domterm.jar
into the executable as initialized data.  But in that case a reasonable
solution is a set of fops to read from main memory, which is trivial.

ldomterm does have a configure option to pre-link the individual files into the
data segment (though I haven't tested that recently), though without compression.
That is also an option.

For now I'll just stick to using lws's builtin zip support (once 2.2 is released and
a Fedora package is available).
	--Per Bothner
per at bothner.com   http://per.bothner.com/

More information about the Libwebsockets mailing list