[Libwebsockets] libwebsockets : Query related to max connections supported, I/O mechanism and message posting

Andy Green andy at warmcat.com
Wed Dec 11 16:20:51 CET 2013

ayush jain <jain.ayush09 at gmail.com> wrote:
>Thanks Andy for clarifying further.
>I saw a buffer with size 4096 bytes during recv() in libwebsockets
>code. Does it mean this is the max message size which we can receive
>for any protocol over websocket using the library? In my case
>application message size can range upto 50k bytes.

No.  But where the browser will make all messages atomic at js layer, lws doesn't.  It'll pass your user code bufferloads as they come.  You can collate it back into an atomic message (of any length) and process it when complete.  websocket 'message' is a logical thing tracked above packets or ws fragments.

But usually, you can find a way to use the pieces with lower latency and no atomic buffer.

You can set the max rx buffer for any chunk in the protocol struct.

>In gist, what I can conclude for integrating libwebsockets in
>multithreading environment is to have our own synchronization
>mechanism (some sort of mutex) while calling API's like
>'libwebsocket_service_fd' (let's say during receiving data) or
>'libwebsocket_callback_on_writable' (let's say during asynchronous
>data to be sent) from different threads. (As both of them are
>referring to same global context)
>Also, there has to be a per connection send/recv buffer required by
>libwebsockets which will be allocated run time (for 1 lakh connection
>if message size limit is 5000bytes, it would be 1 lakh * 5000 bytes ~
>476MB for one side)

it's the right thinking but ws message size is unrelated to the rx buffer.  You'll get the whole message even if GB or TB but it'll come in those size chunks.  And you should think about if you really ever need the atomic message in one place at one time.

Then you can set your protocol's rx buffer size to suit your available memory.


>On 12/9/13, "Andy Green (林安廸)" <andy at warmcat.com> wrote:
>> On 09/12/13 19:49, the mail apparently from ayush jain included:
>>> Hi Andy,
>>> Thanks a lot for your response.
>>> I just had some more queries further inline, would be glad If you
>>> respond them as well.
>>> On 12/5/13, Andy Green <andy at warmcat.com> wrote:
>>>> ayush jain <jain.ayush09 at gmail.com> wrote:
>>>>> Hello everyone,
>>>>> It's been a week or so I started up researching on websockets and
>>>>> landed
>>>>> up here as I found a lot of interesting things about
>>>>> I was seeking some help regarding some aspects about the library
>>>>> terms
>>>>> of performance, I/O mechanism and message posting.
>>>>> 1) Is there any kind of bottleneck or performance issues (if it
>>>>> been
>>>>> tested) in case I wish to integrate the same to a server
>>>>> approx
>>>>> 200k simultaneous tcp/tls connections?
>>>> Well something will always limit performance.  Lws should give very
>>>> performance for one process context it's running in.  Having a
>>>> cpu
>>>> won't help that one process much.
>>>> So if you want strong performance on huge numbers of sockets you
>>>> think about spreading it across multiple instances that have some
>>>> of
>>>> ipc.
>>>>> 2) I see there are 2 ways related to internal/external I/O
>>>>> for
>>>>> integration. But I see the API (libwebsocket_service_fd) takes in
>>>>> pollfd
>>>>> structure as argument, Can I use the libwebsocket library with my
>>>>> existing
>>>>> Epoll implementation when I use external I/O other than POLL, as
>>>>> would be lot of inactve FD's in my case? Or I need to stick to
>>>>> don't have poll mechanism in my server.
>>>> The lib either uses internal poll loop or interfaces to some kind
>>>> external polling loop with an api that uses pollfd semantics.
>>>> There's no requirement the external loop actually is using poll(),
>>>> long
>>>> as you set and interpret the pollfd api parts appropriately that's
>>>> enough.
>>> Let's say am using epoll for tcp handling. Once I accept the
>>> connection using accept(), let's say I register my dataRecv
>> Sorry I have no idea about your "dataRecv callback".
>>> As and when I get read events from EPOLL, my dataRecv() callback
>>> be called which in-turn would call recv() in a stream way with
>>> boundary getting governed by application level protocol.
>>> Since websocket preserves the boundary, I should be able to receive
>>> the message (just like I receive a datagram in UDP) when I get
>>> LWS_CALLBACK_RECEIVE in the callback function for my application
>>> protocol. I should somehow map my FD (on which I was calling recv())
>>> to pollFD structure and call libwebsocket_service_fd() instead of
>>> recv(). Please let me know if my understanding is correct?
>> The last bit is the right idea..
>>   - call some kind of "sleep on a bunch of socket / file descriptors"
>> function somehow,
>>   - if an lws descriptor wants something, fake up a pollfd according
>> what you heard it wanted, and pass it to libwebsocket_service_fd()
>>   - manage that list of descriptors using the extpoll / pollfd stuff,
>> how you really list them for your wait function is not important for
>> as long as you deal with it using fake pollfds
>>>>> 3) When we say libwebsocket is single threaded, Do we mean that
>for all
>>>>> my
>>>>> send and receive using the library, I need to make sure (in case
>>>>> implementing server) that the call to libwebsocket_service_fd()
>and any
>>>>> message posting by requesting callbacks happens in 1 thread? I
>>>>> whatever response my server needs to post back to client after
>>>>> processing
>>>>> of request, How am I going to save or store that data when I get a
>>>>> callback
>>>>> once my server side socket is writeable?
>>>> What it means is that all your interactions with lws must occur
>from the
>>>> user callback code.
>>>> You can allocate per connection user storage, you would do
>>>> like
>>>> manage a per-connection fifo in there and spill it on the writable
>>>> callback
>>>> while it still had something waiting to go.
>>> Thanks for clearing the same. That means that doesn't matter from
>>> which thread am calling, as the sending as well as receiving of data
>>> would be done from user callback code. Whichever thread my control
>>> would be in, would have to request callback so it should take care.
>> Yes, the two restrictons are
>>   - as mentioned, you only do your business from the callback, and by
>> using apis to get a callback when the connection is writeable again,
>> can get optimum behaviours over large numbers of connections.
>> Internally the library is doing the same, if something is not ready
>> would block it asks to get time when the pollfd situation allows (and
>> the rest of the time is spent servicing other sockets or totally
>>   - you need to be consistent about the process context that calls
>> service_fd api.  It should be the same one that did the init and stay
>> the same the whole time.  Otherwise if you forked and call from the
>> other one, it does not have the same per-process data and will break.
>> If you call from a different thread with same data context it might
>> but again there's no locking, so you would have to be consistent
>about it.
>>> As per my understanding, I would have to allocate space per
>>> by specifying 'per_session_data_size' and 'rx_buffer_size' for
>>> as well as receiving. In case the server wish to send out data,
>>> libwebsocket_callback_on_writable() has to be called by saving the
>>> data in 'wsi->user_space', which would be accessible during the
>>> 'LWS_CALLBACK_SERVER_WRITEABLE' callback. While receiving any data,
>>> the same would be something like 'void *in' in the callback during
>>> 'LWS_CALLBACK_RECEIVE' (and the length would be max which we have
>>> specified in 'rx_buffer_size'. Please let me know if my
>>> is correct here?
>> Yeah that's basically it.
>> The callback has a void * "user" pointer which is your per-connection
>> allocation, you cast it to your struct inside your callback and
>> away with just the data you wanted right there immediately.
>> If you follow the thing about asking to get called back when you're
>> writeable you will get very good behavours in terms of keeping pipes
>> stuffed but never stalling, and that will scale to large number of
>> connections with either nearly 0% CPU if idle (you must arrange to
>> the service fd with NULL around once a second to handle timeouts), or
>> busy then low latency consistent service.
>> -Andy
>>>> -Andy
>>>>> I know am asking too much of information here, but would really
>>>>> appreciate
>>>>> if can respond to the same. Looking forward to your response.
>>>>> Thanks in advance.
>>>>> Regards,
>>>>> _Ayush
>>>>> _______________________________________________
>>>>> Libwebsockets mailing list
>>>>> Libwebsockets at ml.libwebsockets.org
>>>>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>>>> --
>>>> Sent from my Android device with K-9 Mail. Please excuse my

More information about the Libwebsockets mailing list