[Libwebsockets] libwebsockets : Query related to max connections supported, I/O mechanism and message posting

ayush jain jain.ayush09 at gmail.com
Wed Dec 11 12:16:22 CET 2013

Thanks Andy for clarifying further.

I saw a buffer with size 4096 bytes during recv() in libwebsockets
code. Does it mean this is the max message size which we can receive
for any protocol over websocket using the library? In my case
application message size can range upto 50k bytes.

In gist, what I can conclude for integrating libwebsockets in
multithreading environment is to have our own synchronization
mechanism (some sort of mutex) while calling API's like
'libwebsocket_service_fd' (let's say during receiving data) or
'libwebsocket_callback_on_writable' (let's say during asynchronous
data to be sent) from different threads. (As both of them are
referring to same global context)

Also, there has to be a per connection send/recv buffer required by
libwebsockets which will be allocated run time (for 1 lakh connection
if message size limit is 5000bytes, it would be 1 lakh * 5000 bytes ~
476MB for one side)

On 12/9/13, "Andy Green (林安廸)" <andy at warmcat.com> wrote:
> On 09/12/13 19:49, the mail apparently from ayush jain included:
>> Hi Andy,
>> Thanks a lot for your response.
>> I just had some more queries further inline, would be glad If you can
>> respond them as well.
>> On 12/5/13, Andy Green <andy at warmcat.com> wrote:
>>> ayush jain <jain.ayush09 at gmail.com> wrote:
>>>> Hello everyone,
>>>> It's been a week or so I started up researching on websockets and I
>>>> landed
>>>> up here as I found a lot of interesting things about libwebsockets.
>>>> I was seeking some help regarding some aspects about the library in
>>>> terms
>>>> of performance, I/O mechanism and message posting.
>>>> 1) Is there any kind of bottleneck or performance issues (if it has
>>>> been
>>>> tested) in case I wish to integrate the same to a server supporting
>>>> approx
>>>> 200k simultaneous tcp/tls connections?
>>> Well something will always limit performance.  Lws should give very good
>>> performance for one process context it's running in.  Having a multicore
>>> cpu
>>> won't help that one process much.
>>> So if you want strong performance on huge numbers of sockets you should
>>> think about spreading it across multiple instances that have some kind
>>> of
>>> ipc.
>>>> 2) I see there are 2 ways related to internal/external I/O mechanism
>>>> for
>>>> integration. But I see the API (libwebsocket_service_fd) takes in
>>>> pollfd
>>>> structure as argument, Can I use the libwebsocket library with my
>>>> existing
>>>> Epoll implementation when I use external I/O other than POLL, as there
>>>> would be lot of inactve FD's in my case? Or I need to stick to POLL? I
>>>> don't have poll mechanism in my server.
>>> The lib either uses internal poll loop or interfaces to some kind of
>>> external polling loop with an api that uses pollfd semantics.
>>> There's no requirement the external loop actually is using poll(), so
>>> long
>>> as you set and interpret the pollfd api parts appropriately that's
>>> enough.
>> Let's say am using epoll for tcp handling. Once I accept the
>> connection using accept(), let's say I register my dataRecv callback.
> Sorry I have no idea about your "dataRecv callback".
>> As and when I get read events from EPOLL, my dataRecv() callback would
>> be called which in-turn would call recv() in a stream way with
>> boundary getting governed by application level protocol.
>> Since websocket preserves the boundary, I should be able to receive
>> the message (just like I receive a datagram in UDP) when I get
>> LWS_CALLBACK_RECEIVE in the callback function for my application
>> protocol. I should somehow map my FD (on which I was calling recv())
>> to pollFD structure and call libwebsocket_service_fd() instead of
>> recv(). Please let me know if my understanding is correct?
> The last bit is the right idea..
>   - call some kind of "sleep on a bunch of socket / file descriptors"
> function somehow,
>   - if an lws descriptor wants something, fake up a pollfd according to
> what you heard it wanted, and pass it to libwebsocket_service_fd()
>   - manage that list of descriptors using the extpoll / pollfd stuff,
> how you really list them for your wait function is not important for lws
> as long as you deal with it using fake pollfds
>>>> 3) When we say libwebsocket is single threaded, Do we mean that for all
>>>> my
>>>> send and receive using the library, I need to make sure (in case am
>>>> implementing server) that the call to libwebsocket_service_fd() and any
>>>> message posting by requesting callbacks happens in 1 thread? I mean
>>>> whatever response my server needs to post back to client after
>>>> processing
>>>> of request, How am I going to save or store that data when I get a
>>>> callback
>>>> once my server side socket is writeable?
>>> What it means is that all your interactions with lws must occur from the
>>> user callback code.
>>> You can allocate per connection user storage, you would do something
>>> like
>>> manage a per-connection fifo in there and spill it on the writable
>>> callback
>>> while it still had something waiting to go.
>> Thanks for clearing the same. That means that doesn't matter from
>> which thread am calling, as the sending as well as receiving of data
>> would be done from user callback code. Whichever thread my control
>> would be in, would have to request callback so it should take care.
> Yes, the two restrictons are
>   - as mentioned, you only do your business from the callback, and by
> using apis to get a callback when the connection is writeable again, you
> can get optimum behaviours over large numbers of connections.
> Internally the library is doing the same, if something is not ready or
> would block it asks to get time when the pollfd situation allows (and
> the rest of the time is spent servicing other sockets or totally asleep)
>   - you need to be consistent about the process context that calls the
> service_fd api.  It should be the same one that did the init and stay
> the same the whole time.  Otherwise if you forked and call from the
> other one, it does not have the same per-process data and will break.
> If you call from a different thread with same data context it might work
> but again there's no locking, so you would have to be consistent about it.
>> As per my understanding, I would have to allocate space per connection
>> by specifying 'per_session_data_size' and 'rx_buffer_size' for sending
>> as well as receiving. In case the server wish to send out data,
>> libwebsocket_callback_on_writable() has to be called by saving the
>> data in 'wsi->user_space', which would be accessible during the
>> 'LWS_CALLBACK_SERVER_WRITEABLE' callback. While receiving any data,
>> the same would be something like 'void *in' in the callback during
>> 'LWS_CALLBACK_RECEIVE' (and the length would be max which we have
>> specified in 'rx_buffer_size'. Please let me know if my understanding
>> is correct here?
> Yeah that's basically it.
> The callback has a void * "user" pointer which is your per-connection
> allocation, you cast it to your struct inside your callback and you're
> away with just the data you wanted right there immediately.
> If you follow the thing about asking to get called back when you're
> writeable you will get very good behavours in terms of keeping pipes
> stuffed but never stalling, and that will scale to large number of
> connections with either nearly 0% CPU if idle (you must arrange to call
> the service fd with NULL around once a second to handle timeouts), or if
> busy then low latency consistent service.
> -Andy
>>> -Andy
>>>> I know am asking too much of information here, but would really
>>>> appreciate
>>>> if can respond to the same. Looking forward to your response.
>>>> Thanks in advance.
>>>> Regards,
>>>> _Ayush
>>>> ------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Libwebsockets mailing list
>>>> Libwebsockets at ml.libwebsockets.org
>>>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>>> --
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.

More information about the Libwebsockets mailing list