[Libwebsockets] Max char limit on websocket requests.

Arsh Patankar arsh.patankar at btinternet.com
Thu Nov 4 17:29:48 CET 2021


I've sorted it now. Thank you very much for the help.

-Arsh

On 11/1/21 5:22 PM, Andy Green wrote:
>
>
> On 11/1/21 15:19, Arsh Patankar wrote:
>
> cc-ing to the list again...
>
>> Been in a back and forth with binance and they've now discovered that 
>> my problem is due to a bug on their servers. They advise to open 
>> multiple 
>
> Well, I did point out that it's always a policy decision of the server 
> if it will give you a 400 or not...
>
>> connections of up to 108 streams (instead of the quoted 200) until 
>> they can fix it. What's the best way to do this using lws? I tried 
>> throwing the same thing on two threads (where each one individually 
>> runs fine) it like this:
>>
>> std::thread websocket1(VWAP::start_websockets, info);
>> std::thread websocket2(VWAP::start_websockets, info);
>>
>> websocket1.join();
>> websocket2.join();
>>
>> But that didn't work.
>
> No... event loops are not threadsafe
>
> https://libwebsockets.org/git/libwebsockets/tree/READMEs/README.event-loops-intro.md 
>
>
> but the good news is you can just use the connection creation info 
> twice to create two connections, ie call lws_client_connect_via_info() 
> twice, I guess modifying the path part inbetwwen to do the other half 
> of whatever it is.
>
> -Andy
>
>> -Arsh
>>
>> On 10/29/21 3:40 PM, Andy Green wrote:
>>>
>>>
>>> On 10/29/21 15:32, Arsh Patankar wrote:
>>>> I've made the changes you said with regards to path and adding the 
>>>> CMake option to the debugger:
>>>>
>>>> I can see now that the cmake debug has been filled with data but 
>>>> non of it shows the request (I scrolled through it all a number of 
>>>> times attached is just an example):
>>>
>>> When it says "Manually-specified variables were not used by the 
>>> project" it means cmake could not wire up what you gave to an option 
>>> in the CMakeLists.txt.
>>>
>>> But, that has been an option in lws for a while
>>>
>>> https://libwebsockets.org/git/libwebsockets/tree/CMakeLists.txt#n194
>>>
>>> either the lws is too old to have that option, or something to do 
>>> with your build process; I guess you're giving the cmake option to 
>>> the outer project.
>>>
>>> What I would do is hack the couple of lines this enables in, this is 
>>> the place
>>>
>>> https://libwebsockets.org/git/libwebsockets/tree/lib/tls/openssl/openssl-ssl.c#n343-351 
>>>
>>>
>>> -Andy
>>>
>>>> The standard debugger isn't showing anything useful either:
>>>>
>>>> the path is, however, correct. Just to show you, here is an example 
>>>> of it working when the request is smaller:
>>>>
>>>> Have I done what you said correctly?
>>>>
>>>> -Arsh
>>>>
>>>> On 10/29/21 7:12 AM, Andy Green wrote:
>>>>>
>>>>>
>>>>> On 10/28/21 21:15, Arsh Patankar wrote:
>>>>>
>>>>>> I've found the bit of code that sets that up now. The relavent 
>>>>>> one I believe is this one:
>>>>>>
>>>>>> void binance::Websocket::init()
>>>>>> {
>>>>>>     lws_context_creation_info info;
>>>>>>     memset(&info,0,sizeof(info));
>>>>>>
>>>>>>     info.port =CONTEXT_PORT_NO_LISTEN;
>>>>>>     info.protocols = protocols;
>>>>>>     info.gid = -1;
>>>>>>     info.uid = -1;
>>>>>>      info.pt_serv_buf_size=4096*5;
>>>>>>      info.max_http_header_data =1024*20;
>>>>>>
>>>>>>     // This option is needed here to imply 
>>>>>> LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT // option, which must be set 
>>>>>> on newer versions of OpenSSL. info.options 
>>>>>> =LWS_SERVER_OPTION_REQUIRE_VALID_OPENSSL_CLIENT_CERT;
>>>>>>
>>>>>>     context = lws_create_context(&info);
>>>>>> }
>>>>>
>>>>> Yes that looks like the one.
>>>>>
>>>>>> This function also exists but i don't think it is the limiting 
>>>>>> factor:
>>>>>>
>>>>>> void binance::Websocket::connect_fendpoint(CB cb,const char* path)
>>>>>> {
>>>>>> binance::Logger::write_log("<connect_endpoint>");
>>>>>>
>>>>>>      char ws_path[1024*20];// was 1024 strcpy(ws_path, path);
>>>>>
>>>>> So the strcpy() is a buffer overflow, but since the path is not 
>>>>> modified, the whole ws_path[] thing and the strcpy can go away and 
>>>>> just set ccinfo.path to path itself. Copying it into a stack 
>>>>> buffer doesn't do anything except crash your program if it's 
>>>>> bigger than the buffer. Internally lws handles it by a mixture of 
>>>>> preallocated buffers (that it carefully size checks) and a 
>>>>> custom-sized heap buffer to hold it in case of need later for 
>>>>> redirects.
>>>>>
>>>>>>      // Connect if we are not connected to the server. 
>>>>>> lws_client_connect_info ccinfo = {0 };
>>>>>>      ccinfo.context = context;
>>>>>>      ccinfo.address =BINANCE_FWS_HOST;
>>>>>>      ccinfo.port =BINANCE_FWS_PORT;
>>>>>>      ccinfo.path = ws_path;
>>>>>>      ccinfo.host = lws_canonical_hostname(context);
>>>>>>      ccinfo.origin ="origin";
>>>>>>      ccinfo.protocol = protocols[0].name;
>>>>>>      ccinfo.ssl_connection =LCCSCF_USE_SSL 
>>>>>> |LCCSCF_ALLOW_SELFSIGNED |LCCSCF_SKIP_SERVER_CERT_HOSTNAME_CHECK;
>>>>>>      lws* conn = lws_client_connect_via_info(&ccinfo);
>>>>>>      handles[conn] = cb;
>>>>>> }
>>>>>>
>>>>>> Also just noticed a different error (was hidden in the logs) 
>>>>>> 'LWS_CALLBACK_CLIENT_CONNECTION_ERROR'. Not sure if this gives a 
>>>>>> clue into what is causing the problem. I've added the 
>>>>>> 'info.pt_serv_buf_size=4096*5;' but that hasn't fixed it.
>>>>>
>>>>> If there's more than one problem, not finding yourself suddenly at 
>>>>> the top of the mountain doesn't change you are no longer at the 
>>>>> bottom of the mountain.  For example the strcpy [1024] thing alone 
>>>>> would kill you and I can't guess the existence of that.  What it 
>>>>> means is [y]our model of what is really happening, doesn't reflect 
>>>>> reality yet.
>>>>>
>>>>> Lws works with, and tests for operation with in CI, huge urls (2KB 
>>>>> path)
>>>>>
>>>>> https://libwebsockets.org/git/libwebsockets/tree/minimal-examples-lowlevel/http-client/minimal-http-client-hugeurl 
>>>>>
>>>>>
>>>>> so this is not new ground and as far as that goes, it is supported.
>>>>>
>>>>> There's a convenient, simple (if noisy) debug helper cmake option 
>>>>> for lws, -DLWS_TLS_LOG_PLAINTEXT_TX=1, this will hexdump what lws 
>>>>> is sending on the tls connection just before the data is 
>>>>> encrypted.  So it will show you what we are actually trying to 
>>>>> send to the server.  Unless the server decides to 400 us for its 
>>>>> own policy reasons, whatever is wrong should show up in there, 
>>>>> even if the root cause is earlier.
>>>>>
>>>>> -Andy
>>>>>
>>>>>> -Arsh
>>>>>>
>>>>>> On 10/27/21 3:29 PM, andy at warmcat.com wrote:
>>>>>>> On 27 October 2021 14:25:53 UTC, Arsh 
>>>>>>> Patankar<arsh.patankar at btinternet.com> wrote:
>>>>>>>> You are correct, what I am using came with lws. Following
>>>>>>>> 'pt_serv_buf_size', I find usages in the file 'context.c'.
>>>>>>> ... no, you do not need to change lws.
>>>>>>>
>>>>>>> Some other software, outside of lws, is creating an lws context 
>>>>>>> using one of those info structs.  Find and modify that before it 
>>>>>>> calls through to lws to create the context.
>>>>>>>
>>>>>>> -Andy


More information about the Libwebsockets mailing list