[Libwebsockets] libwebsockets : Query related to max connections supported, I/O mechanism and message posting

"Andy Green (林安廸)" andy at warmcat.com
Mon Dec 9 13:04:40 CET 2013

On 09/12/13 19:49, the mail apparently from ayush jain included:
> Hi Andy,
> Thanks a lot for your response.
> I just had some more queries further inline, would be glad If you can
> respond them as well.
> On 12/5/13, Andy Green <andy at warmcat.com> wrote:
>> ayush jain <jain.ayush09 at gmail.com> wrote:
>>> Hello everyone,
>>> It's been a week or so I started up researching on websockets and I
>>> landed
>>> up here as I found a lot of interesting things about libwebsockets.
>>> I was seeking some help regarding some aspects about the library in
>>> terms
>>> of performance, I/O mechanism and message posting.
>>> 1) Is there any kind of bottleneck or performance issues (if it has
>>> been
>>> tested) in case I wish to integrate the same to a server supporting
>>> approx
>>> 200k simultaneous tcp/tls connections?
>> Well something will always limit performance.  Lws should give very good
>> performance for one process context it's running in.  Having a multicore cpu
>> won't help that one process much.
>> So if you want strong performance on huge numbers of sockets you should
>> think about spreading it across multiple instances that have some kind of
>> ipc.
>>> 2) I see there are 2 ways related to internal/external I/O mechanism
>>> for
>>> integration. But I see the API (libwebsocket_service_fd) takes in
>>> pollfd
>>> structure as argument, Can I use the libwebsocket library with my
>>> existing
>>> Epoll implementation when I use external I/O other than POLL, as there
>>> would be lot of inactve FD's in my case? Or I need to stick to POLL? I
>>> don't have poll mechanism in my server.
>> The lib either uses internal poll loop or interfaces to some kind of
>> external polling loop with an api that uses pollfd semantics.
>> There's no requirement the external loop actually is using poll(), so long
>> as you set and interpret the pollfd api parts appropriately that's enough.
> Let's say am using epoll for tcp handling. Once I accept the
> connection using accept(), let's say I register my dataRecv callback.

Sorry I have no idea about your "dataRecv callback".

> As and when I get read events from EPOLL, my dataRecv() callback would
> be called which in-turn would call recv() in a stream way with
> boundary getting governed by application level protocol.
> Since websocket preserves the boundary, I should be able to receive
> the message (just like I receive a datagram in UDP) when I get
> LWS_CALLBACK_RECEIVE in the callback function for my application
> protocol. I should somehow map my FD (on which I was calling recv())
> to pollFD structure and call libwebsocket_service_fd() instead of
> recv(). Please let me know if my understanding is correct?

The last bit is the right idea..

  - call some kind of "sleep on a bunch of socket / file descriptors" 
function somehow,

  - if an lws descriptor wants something, fake up a pollfd according to 
what you heard it wanted, and pass it to libwebsocket_service_fd()

  - manage that list of descriptors using the extpoll / pollfd stuff, 
how you really list them for your wait function is not important for lws 
as long as you deal with it using fake pollfds

>>> 3) When we say libwebsocket is single threaded, Do we mean that for all
>>> my
>>> send and receive using the library, I need to make sure (in case am
>>> implementing server) that the call to libwebsocket_service_fd() and any
>>> message posting by requesting callbacks happens in 1 thread? I mean
>>> whatever response my server needs to post back to client after
>>> processing
>>> of request, How am I going to save or store that data when I get a
>>> callback
>>> once my server side socket is writeable?
>> What it means is that all your interactions with lws must occur from the
>> user callback code.
>> You can allocate per connection user storage, you would do something like
>> manage a per-connection fifo in there and spill it on the writable callback
>> while it still had something waiting to go.
> Thanks for clearing the same. That means that doesn't matter from
> which thread am calling, as the sending as well as receiving of data
> would be done from user callback code. Whichever thread my control
> would be in, would have to request callback so it should take care.

Yes, the two restrictons are

  - as mentioned, you only do your business from the callback, and by 
using apis to get a callback when the connection is writeable again, you 
can get optimum behaviours over large numbers of connections. 
Internally the library is doing the same, if something is not ready or 
would block it asks to get time when the pollfd situation allows (and 
the rest of the time is spent servicing other sockets or totally asleep)

  - you need to be consistent about the process context that calls the 
service_fd api.  It should be the same one that did the init and stay 
the same the whole time.  Otherwise if you forked and call from the 
other one, it does not have the same per-process data and will break. 
If you call from a different thread with same data context it might work 
but again there's no locking, so you would have to be consistent about it.

> As per my understanding, I would have to allocate space per connection
> by specifying 'per_session_data_size' and 'rx_buffer_size' for sending
> as well as receiving. In case the server wish to send out data,
> libwebsocket_callback_on_writable() has to be called by saving the
> data in 'wsi->user_space', which would be accessible during the
> 'LWS_CALLBACK_SERVER_WRITEABLE' callback. While receiving any data,
> the same would be something like 'void *in' in the callback during
> 'LWS_CALLBACK_RECEIVE' (and the length would be max which we have
> specified in 'rx_buffer_size'. Please let me know if my understanding
> is correct here?

Yeah that's basically it.

The callback has a void * "user" pointer which is your per-connection 
allocation, you cast it to your struct inside your callback and you're 
away with just the data you wanted right there immediately.

If you follow the thing about asking to get called back when you're 
writeable you will get very good behavours in terms of keeping pipes 
stuffed but never stalling, and that will scale to large number of 
connections with either nearly 0% CPU if idle (you must arrange to call 
the service fd with NULL around once a second to handle timeouts), or if 
busy then low latency consistent service.


>> -Andy
>>> I know am asking too much of information here, but would really
>>> appreciate
>>> if can respond to the same. Looking forward to your response.
>>> Thanks in advance.
>>> Regards,
>>> _Ayush
>>> ------------------------------------------------------------------------
>>> _______________________________________________
>>> Libwebsockets mailing list
>>> Libwebsockets at ml.libwebsockets.org
>>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.

More information about the Libwebsockets mailing list