[Libwebsockets] Unable to send very large sized packets through websockets
"Andy Green (林安廸)"
andy at warmcat.com
Wed Nov 13 01:29:38 CET 2013
On 12/11/13 01:57, the mail apparently from Hemant Kumar included:
> Hi Andy
> My initial server log shows as follows:
> [root at hkumar-linux exp]# ./wserver [1384192143:7875] NOTICE: Initial
> logging level 15 [1384192143:7876] NOTICE: Initial logging level 15
> [1384192143:7876] NOTICE: Library version: 1.3 5dc62ea
> [1384192143:7876] INFO: LWS_MAX_HEADER_LEN: 1024 [1384192143:7876]
> INFO: LWS_MAX_PROTOCOLS: 5 [1384192143:7876] INFO:
> LWS_MAX_EXTENSIONS_ACTIVE: 3 [1384192143:7876] INFO:
> SPEC_LATEST_SUPPORTED: 13 [1384192143:7876] INFO: AWAITING_TIMEOUT:
> 5 [1384192143:7876] INFO: SYSTEM_RANDOM_FILEPATH: '/dev/urandom'
> [1384192143:7876] INFO: LWS_MAX_ZLIB_CONN_BUFFER: 65536
> [1384192143:7877] NOTICE: Started with daemon pid 0
> [1384192143:7877] NOTICE: static allocation: 131448 + (16 x 65535
> fds) = 1180008 bytes [1384192143:7880] NOTICE: canonical_hostname =
> hkumar-linux [1384192143:7880] NOTICE: Compiled without SSL support
> [1384192143:7880] NOTICE: per-conn mem: 160 + 1360 headers +
> protocol rx buf [1384192143:7880] INFO: insert_wsi_socket_into_fds:
> wsi=0x197d010, sock=4, fds pos=0 [1384192143:7880] NOTICE: Listening
> on port 80 [1384192143:7880] NOTICE: starting server... starting
> As you can see version is 1.3. I am running client-server on local
> machine only and I guess compression extension is not active. Nothing
> wrong in sending data in chunks of 32K/64K, but I was trying to get
> the maximum pkt size we can send in one go. What is the actual
> concept behind LWS_MAX_SOCKET_IO_BUF and why it cant be set to
> unusually high value as 1024K,can you please clarify on this ? We can
If you don't let lws know in your protocol definition what is the max
size you expect to send is (using .rx_buffer_size), it defaults to
When the connection is made, lws arranges with the OS to reserve at
least that amount of buffer space for the connection.
You should better leave LWS_MAX_SOCKET_IO_BUF alone and set
your_protocol.rx_buffer_size to something suitable, maybe 32K or 64K if
it will let you have it.
Then at least as I observe, you won't get signalled the socket is
writable again until you can write another chunk of
So send a chunk like that and ask to be called back when you can send
more (if you can already send more, you'll get called straight back to
do it again). If the link is slow, it can service other sockets until
that link can accept more and call you back then.
> send large websocket fragments, but then why I am maxing out
> depending upon value set in for IO buffer above and seems like there
> is a limit on that.
There is always anyway a limit to how much the OS will take off your
hands during a socket send. That limit depends on memory pressure on
the system and how quickly the receiving peer is draining the buffer.
But you can never pass like 10MBytes or something to the OS to deal with
in one send / write. lws is just reflecting that.
Also once you have stuffed the pipe, so long as latency for coming back
and shoving more in it is low (as it should be with lws) the effect on
the ability for the network device to send a full frame or send it
quickly should anyway be minimal.
> Regards Hemant
> ________________________________________ From: Andy Green
> [extracats at googlemail.com] on behalf of "Andy Green (林安廸)"
> [andy at warmcat.com] Sent: Saturday, November 09, 2013 5:09 PM To:
> Hemant Kumar; libwebsockets at ml.libwebsockets.org Subject: Re:
> [Libwebsockets] Unable to send very large sized packets through
> On 10/11/13 08:29, the mail apparently from Hemant Kumar included:
>> I have set the size of LWS_MAX_SOCKET_IO_BUF to 64K so that I can
>> send large sized packets. If I send 32 K packets, my callback
>> handler is called twice as expected , but as I tend to approach 64K
>> ,it is called every once for the packet send: [1384042703:8452]
>> NOTICE: 0 997 (0) :tx 60016 <===sending 60K sized pkts
>> [1384042703:8453] NOTICE: 1 998 (0) :tx 60016 [1384042703:8458]
>> NOTICE: LWS_CALLBACK_CLIENT_WRITEABLE [1384042703:8458] NOTICE: 0
>> 999 (0) :tx 60016 [1384042703:8464] NOTICE:
>> LWS_CALLBACK_CLIENT_WRITEABLE [1384042703:8464] NOTICE: 0 1000 (0)
>> :tx 60016 [1384042703:8470] NOTICE: LWS_CALLBACK_CLIENT_WRITEABLE
>> When I try sending 63K packet , it fails sending 9th packet with
>> following error:
>> [1384042832:4989] NOTICE: 1 8 (0) :tx 63016 [1384042832:4991]*ERR:
>> Unable to spill ext 63008 vs 49152*
> You have a compression extension active on this connection?
> What version of lws are you using?
>> [1384042832:4991] NOTICE: 2 9 (0) :tx -1
>> To get over this issue, I increased the LWS_MAX_SOCKET_IO_BUF to
>> 128K. In this case as I approach 110K, it fails as:
>> [1384043094:9284] NOTICE: LWS_CALLBACK_CLIENT_WRITEABLE
>> [1384043094:9286] NOTICE: 0 43 (0) :tx 110028 [1384043094:9296]
>> NOTICE: LWS_CALLBACK_CLIENT_WRITEABLE [1384043094:9299] NOTICE: 0
>> 44 (0) :tx 110028 [1384043094:9300]*ERR: Unable to spill ext 110014
>> vs 102978* [1384043094:9300] NOTICE: 1 45 (0) :tx -1
>> [1384043094:9300] INFO: libwebsocket_service_fd: closing
>> [1384043094:9301] NOTICE: LWS_CALLBACK_CLOSED
>> What is the maximum limit we can set on LWS_MAX_SOCKET_IO_BUFand
>> what is the relation with the max size packet we can send? When I
>> tried setting this to 1024K, I was not able to send a single
> What's wrong with sending your content in 32K chunks?
>> Can you please clarify and suggest way for sending very large
>> sized packet without application worrying about fragmentation ?
>> Websocket as the RFC states is supposed to be message oriented
>> protocol, so I believe there has to be a way not to care about frag
>> issue associated with large sized packets?
> The RFC explains how to use websocket fragments (not tcp fragments)
> to send as much as you like, even endless streams.
>> Thanks Hemant
>> _______________________________________________ Libwebsockets
>> mailing list Libwebsockets at ml.libwebsockets.org
More information about the Libwebsockets