[Libwebsockets] Using websockets as a pure buffer manipulation engine.

Alan Conway aconway at redhat.com
Thu Nov 17 15:50:31 CET 2016

On Thu, 2016-11-17 at 22:04 +0800, Andy Green wrote:
> > I'm working on qpid.apache.org/proton, an AMQP messaging library.
> > It
> > currently has a single-threaded, event-based "reactor" API using
> > sockets & poll. It also has a low-level "protocol driver" that
> > deals
> > strictly with byte buffers in memory - no IO assumptions.
> I understand the kind of abstraction you mean, but I think it's not
> enough for a practical system without also considering flow control
> (equivalent to POLLIN/POLLOUT)

Ohhhh yes indeedy. If I had a dime for every distributed system screwup
I've fixed (or caused) due to ignoring flow control, I'd have a lot of
dimes :)

Here's how it works for proton: 2 variable sized but bounded buffers
for read and write (they are actually variable sized windows on a fixed
buffer). Normal operation is something like:

- non-blocking or async read of max read-buffer-size bytes
- parse & handle events which puts stuff in the write-buffer
- non-blocking or async partial write of write-buffer till it's empty

EPOLLIN == read-buffer-size > 0
EPOLLOUT == write-buffer-size > 0

If the write buffer fills up, the read-buffer is disabled till there is
some write space.

There's definitely room for improvement on the buffering scheme but it
does handle the flow control problem pretty well.

> > The FD notion also isn't very helpful for fast native windows IOCP
> > - we
> > ended up having to fake a poll()-like loop which is a terribly
> > inefficient way to use IOCP.
> Dunno what iocp is, but --->

Windows IO Completion Ports: a multi-threaded, proactive IO framework.
Most apps deal with it by making it look like a single-threaded poller
(e.g. libuv) which gives acceptable single-threaded performance but
does not scale as well on multi-cores as proper multi-threaded use.

> > So I'd love to see a byte-buffer based websocket "driver" that
> > could be
> > used independently of the libwebsocket loop and has no assumptions
> The 'flow control' type concerns are not theoretical... 

+1000, preaching to the choir. Beware unbounded data structures, there
are no computers with unbounded memory ;)

> Sure, it should not be that hard since the recv() and send() bits are
> already isolated.  But you will need a flow control method even to
> duplicate the mirror demo reliably.

Yep. What do you think of the scheme described above? There are other
ways to skin that cat. We know proton's buffer scheme has some
limitations (control of allocation, may force copies from externally-
owned buffers) but it has the advantage of being simple. If you want to
come up with something more sophisticated I'd enjoy that discussion.

More information about the Libwebsockets mailing list