[Libwebsockets] Using websockets as a pure buffer manipulation engine.

Andy Green andy at warmcat.com
Thu Nov 17 15:04:26 CET 2016

On November 17, 2016 9:45:03 PM GMT+08:00, Alan Conway <aconway at redhat.com> wrote:
>On Wed, 2016-11-16 at 10:23 +0800, Andy Green wrote:
>> On Wed, 2016-11-16 at 08:46 +0800, Andy Green wrote:
>> > 
>> > On Tue, 2016-11-15 at 14:59 -0500, Alan Conway wrote:
>> > 
>> > > 
>> > > I had a look in docs etc. but cound't find an answer for this:
>> > > 
>> > > I want to add websocket support to a project that already has
>> > > it's
>> > > own
>> > > multi-platform proactor, multi-SSL support etc. I would like to
>> > > use
>> > > libwebsockets as a pure codec, dealing with just byte buffers
>> > > from
>> > > my
>> > > own IO layer (which might not even use file descriptors in some
>> > > cases).
>> > > Is that possible?
>> > 
>> > As it is, no.  Lws can hook into someone else's existing event loop
>> > okay, and many people have done that over the years, but the lowest
>> > common denominator as it stands is a socketfd.
>> > 
>> > Inside lws the stuff that actually touches the fds is isolated in a
>> > few
>> > places and eg, pure external event loops are supported like libuv
>> > that
>> > expose wrapped platform-specific objects instead of fds.  So it's
>> > not
>> > out of the question to modify it to allow it.
>> > 
>> > Still the most important event for lws is that the connection to
>> > the
>> > peer is "writeable", eg, POLLOUT from poll().  In all the ways to
>> > use
>> > lws this provides the basic heartbeat for the sequencing.  Is that
>> > an
>> > event that fits into your picture?
>> ... and thinking about it RX flow control management, that lws
>> controls
>> whether pending RX packets remain unread by POLLIN control or other
>> similar scheme, is also indispensable for a practical system.
>Thanks! I have a short term project where this will work but a longer
>term one that is more interesting. 
>I'm working on qpid.apache.org/proton, an AMQP messaging library. It
>currently has a single-threaded, event-based "reactor" API using
>sockets & poll. It also has a low-level "protocol driver" that deals
>strictly with byte buffers in memory - no IO assumptions.

I understand the kind of abstraction you mean, but I think it's not enough for a practical system without also considering flow control (equivalent to POLLIN/POLLOUT)

>That can be used to build more interesting, multi-threaded apps. For
>example we have a concurrent Go binding that manages native Go net.Conn
>connections and pumps data thru the driver. Go has no notion of
>"polling" and net.Conn connections don't have an FD (some do deep
>inside, but e.g. in memory pipes don't at all)

Consider something like the mirror protocol demo.

There is a network programming primitive here which requires that data sources are subordinate to the sinks.  It's directly related to the same problem in shell pipes.  If you type ^S in the console we must flow control the source.

 - regulation of the data output flow must reflect the state of the peer receiver

 - regulation of accepting new input, must be contingent on the output side willing to accept new input, if there is a path input -> output

>The FD notion also isn't very helpful for fast native windows IOCP - we
>ended up having to fake a poll()-like loop which is a terribly
>inefficient way to use IOCP.

Dunno what iocp is, but --->

>So I'd love to see a byte-buffer based websocket "driver" that could be
>used independently of the libwebsocket loop and has no assumptions

The 'flow control' type concerns are not theoretical... to work in a practical system they have to be satisfied even if they are called something different.

>about FDs. Is this a feature you would consider? I would be happy to
>contribute (with help!) as I've just gone thru this process with proton
>so I have some experience :) It is not as drastic as it might sound -
>the "driver" is still event-based, it is simply being pumped bytes from
>some IO-aware code instead of being directly coupled to that code.

Sure, it should not be that hard since the recv() and send() bits are already isolated.  But you will need a flow control method even to duplicate the mirror demo reliably.



More information about the Libwebsockets mailing list