[Libwebsockets] polling for file descriptor i/o
andy at warmcat.com
Mon Mar 6 20:27:46 CET 2017
On March 7, 2017 3:03:34 AM GMT+08:00, Per Bothner <per at bothner.com> wrote:
>On 03/05/2017 10:48 PM, Andy Green wrote:
>>> The uri in the call to lws_http_serve does not start with a '/',
>which may be the issue:
>> I think I found it, please refetch master.
>> I also did some valgrind and coverity testing, the latest master has
>those fixes too.
>It works. Thank you.
>However, it appears that every time lws_fops_zip_open is called, it
>zipfile afresh, and *rescans* the central directory.
>This seems very unfortunate. In my implementation, the zipfile was
>opened and its directory scanned just once.
Yeah... but your implementation allocates an unbounded amount of memory then according to the directory size for an unbounded period (and allocates an unbounded amount of memory inflating the whole content into memory before serving it). These characteristics mean that the resource-constrained case that can benefit most from this can't use it, especially platforms that cannot mmap the zip as it required.
In lws implementation the total heap usage serving from inside a zip is restricted to a few hundred bytes including the decompress block buffer; the device can cheaply serve huge files from the zip bigger than its own memory, compressed or decompressed. Even the weakest platforms can afford to use it. And it uses platform fops only to access the zip file without needing mmap.
>Of course one option is I could keep my mmap-based zip-reader
>implementation and just
>ignore the one in libwebsockets. The amount of "code duplication"
>would be trivial.
It'd be a shame... once the zip has been read once, in Linux those pieces will be in memory in the block cache. Seeking and reading from there is already doing it from ram without going back to the storage device.
If some specific thing you need is objectively slow, we can discuss targeting that with a CMake option indicating on that platform, speed is more important than memory usage. But I suggest measuring it and identifying what is too slow first so we discuss the right thing.
More information about the Libwebsockets