[Libwebsockets] libwebsockets-1.4 HTTP server help with huge http chunks

Andy Green andy at warmcat.com
Thu Oct 15 04:20:09 CEST 2015



On 7 July 2015 14:10:23 GMT+09:00, Ash 20001 <ash20001 at hotmail.com> wrote:
>Also on other thing to note: the windows implementation of buffered
>readers does not even seem to exist (the linked list looping and
>marking POLLIN)...and the problem exists there as well.
>
>From: andy.green at linaro.org
>To: andy.green at linaro.org
>Date: Mon, 6 Jul 2015 22:08:51 -0700
>CC: libwebsockets at ml.libwebsockets.org
>Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help with
>huge http chunks
>
>
>
>
>I think I found the problem. In lws-plat-unix, where you go through the
>pending read list and fake POLLIN status, you use wsi->sock, which is
>incorrect, it should use the index that corresponds to the context->fds
>array index.

Thanks... this got fixed in a2a4b0b0 by Thomas Greenslade in the meanwhile.

-Andy

>So this code seems to work for me, but I have no tested it thoroughly
>yet. (excuse the formatting)
>#ifdef LWS_OPENSSL_SUPPORT        /*         * For all guys with
>buffered SSL read data already saved up, if they         * are not
>flowcontrolled, fake their POLLIN status so they'll get         *
>service to use up the buffered incoming data, even though their        
>* network socket may have nothing         */
>wsi = context->pending_read_list;        while (wsi) {          for (n
>= 0; n < context->fds_count; n++) {            if (context->fds[n].fd
>== wsi->sock)              { index = n; break;}          }
>                wsi_next = wsi->pending_read_list_next;
>context->fds[index].revents |=                                         
>context->fds[index].events & POLLIN;                if
>(context->fds[index].revents & POLLIN) {                        /*     
>* he's going to get serviced now, take him off the                     
>* list of guys with buffered SSL.  If he still has some                
>* at the end of the service, he'll get put back on the                 
>* list then.                         */                       
>lws_ssl_remove_wsi_from_buffered_list(context, wsi);                }  
>             wsi = wsi_next;        }#endif
>
>
>From: andy.green at linaro.org
>To: andy.green at linaro.org
>Date: Mon, 6 Jul 2015 11:55:15 -0700
>CC: libwebsockets at ml.libwebsockets.org
>Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help with
>huge http chunks
>
>
>
>
>Hello Roger,After spending more time debugging this issue, it looks
>like the data is stuck somewhere in the kernel. From what I see after
>the first 4096 packet is received, libwebsockets just goes into its
>normal 50 ms service loop with no indication that data is still pending
>to be read. During this time, the post request on the client is sitting
>waiting for the server to complete receiving all the data. But if I
>cancel the request (I used Postman for this) on the client side, the
>libwebsockets server side immediately gets the rest of the data. 
>However, I have to cancel within around 5 seconds (maybe some timeout
>somewhere) to get the data.
>Does this make any sense to you in diagnosing the problem?
>From: ash20001 at hotmail.com
>To: andy.green at linaro.org
>CC: libwebsockets at ml.libwebsockets.org
>Subject: RE: [Libwebsockets] libwebsockets-1.4 HTTP server help with
>huge http chunks
>Date: Thu, 2 Jul 2015 14:24:05 -0700
>
>
>
>
>Hello Roger,Thank you for your investigation on this! I also found this
>same problem where the next wsi and the current wsi is the same and put
>in a fix myself for it but it is still not solving the issue. It is
>almost as if the SSL socket has no more read data in it even though it
>was sent. 
>Here is an excerpt of the log when posting around 9K bytes of data to
>the server:
>lwsts[13369]: known hdr 8lwsts[13369]: libwebsocket_parse sees parsing
>completelwsts[13369]: libwebsocket_ensure_user_space: 0x8971fc0
>protocol 0x8063340lwsts[13369]: Method: POST request for '/'    post  =
>/    host: = 105.145.65.81:7681    connection: = keep-alive    origin:
>= chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop    http/1.1  =
>HTTP/1.1    accept: = */*    accept-encoding: = gzip, deflate   
>accept-language: = en-US,en;q=0.8    cache-control: = no-cache   
>cookie: = test=LWS_1435790394_190834_COOKIE    content-length: = 9438  
>content-type: = text/plain;charset=UTF-8    user-agent: = Mozilla/5.0
>(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)
>Chrome/43.0.2357.130 Safari/537.36lwsts[13369]: libwebsocket_read:
>read_oklwsts[13369]: LWS_CALLBACK_HTTP_BODY: asdasdasdasasdasdasd...
>len 4096lwsts[13369]: libwebsocket_read: read_ok
>
>As you can see the first 4k is received (with the body being
>'asdasd...') and then it just spins in the main service loop.
>> From: andy.green at linaro.org
>> Date: Thu, 2 Jul 2015 20:41:13 +0100
>> To: andy.green at linaro.org
>> CC: libwebsockets at ml.libwebsockets.org
>> Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help with
>huge	http chunks
>> 
>> Hi Ash,
>> 
>> I think I've found your problem, there is an attempt at a fix here:
>> 
>>
>https://github.com/warmcat/libwebsockets/commit/fc10940ec1c1e974a94eb5ed171f1aedaafef93e
>> 
>> Cheers,
>> 
>> Roger
>> 
>> 
>> 
>> On Thu, Jul 2, 2015 at 6:42 PM, Ash 20001 <andy.green at linaro.org>
>wrote:
>> > I have confirmed this issue exists on all versions of websockets >
>1.3 and
>> > on the windows builds besides linux builds.
>> >
>> > This sounds like a bug in libwebsockets when using HTTP protocol
>over HTTPS
>> > on the server side.
>> > If anyone has any insight into how to fix the problem, it would be
>greatly
>> > appreciated.
>> >
>> > ________________________________
>> > From: andy.green at linaro.org
>> > To: libwebsockets at ml.libwebsockets.org
>> > Date: Tue, 30 Jun 2015 15:19:10 -0700
>> >
>> > Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help
>with huge
>> > http chunks
>> >
>> > Sorry for the other emails, I have much easier reproducible steps
>for this
>> > that maybe you guys can help me with:
>> >
>> > 1. Build 1.4 test-server binary.
>> > 2. Run it with --ssl option.
>> > 3. Use an HTTP client on another machine, like Advanced Rest Client
>to send
>> > a POST data to https://<your server>:<your ssl port>. Start with
>under 4k
>> > sized payloads and notice it works and server receives both BODY
>and BODY
>> > COMPLETION.
>> > 4. Change to higher than 4k POST data payload, and notice that the
>server
>> > only receives the first 4k chunk and then gets stuck.
>> >
>> >
>> >
>> > ________________________________
>> > From: andy.green at linaro.org
>> > To: libwebsockets at ml.libwebsockets.org
>> > Date: Tue, 30 Jun 2015 00:53:59 -0700
>> > Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help
>with huge
>> > http chunks
>> >
>> > One thing I forgot to mention: this only happens on HTTPS, not
>HTTP. HTTP
>> > works fine. Any ideas why HTTPS would be doing this? It gets the
>first 4K
>> > data just fine, just hangs after that and never gets the second or
>more 4K
>> > data.
>> >
>> > ________________________________
>> > From: andy.green at linaro.org
>> > To: libwebsockets at ml.libwebsockets.org
>> > Date: Mon, 29 Jun 2015 22:08:58 -0700
>> > Subject: [Libwebsockets] libwebsockets-1.4 HTTP server help with
>huge http
>> > chunks
>> >
>> > Hello,
>> > I am trying to host an HTTP server (using the test-server.c sample
>code). I
>> > have a client app that sends big chunks of HTTP data (around
>10k-15k) to the
>> > server.
>> > The server doesn't seem to parse the body fragments well.
>> >
>> > By default libwebsockets delivers data in 4k chunks. This seems to
>work fine
>> > for chunks less than 8k after adding
>libbwesockets_return_http_status with
>> > HTTP_OK after each LWS_CALLBACK_HTTP_BODY message. In the case of
>less than
>> > 8K and greater than 4K, I get two LWS_CALLBACK_HTTP_BODY message,
>the first
>> > with 4k and then the remaining amount in the next message. Then I
>received
>> > the LWS_CALLBACK_HTTP_BODY_COMPLETION message.
>> >
>> > But If I try something bigger than 8k, I never get the 3rd or
>higher chunk
>> > in the BODY Message, and sometimes the whole client app errors out.
> I
>> > cannot find any sample out there to properly handle large HTTP
>chunks. I am
>> > hoping maybe you guys might have something out there.
>> >
>> > Also, changing the rx_buffer_size in the http protocol seems to
>make no
>> > difference.
>> >
>> > Thanks for any help!!
>> >
>> > _______________________________________________ Libwebsockets
>mailing list
>> > Libwebsockets at ml.libwebsockets.org
>> > http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>> >
>> > _______________________________________________ Libwebsockets
>mailing list
>> > Libwebsockets at ml.libwebsockets.org
>> > http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>> >
>> > _______________________________________________ Libwebsockets
>mailing list
>> > Libwebsockets at ml.libwebsockets.org
>> > http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>> >
>> > _______________________________________________
>> > Libwebsockets mailing list
>> > Libwebsockets at ml.libwebsockets.org
>> > http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>> >
>> _______________________________________________
>> Libwebsockets mailing list
>> Libwebsockets at ml.libwebsockets.org
>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
> 		 	   		   		 	   		  
>
>_______________________________________________
>Libwebsockets mailing list
>Libwebsockets at ml.libwebsockets.org
>http://ml.libwebsockets.org/mailman/listinfo/libwebsockets 		 	   		  
>
>_______________________________________________
>Libwebsockets mailing list
>Libwebsockets at ml.libwebsockets.org
>http://ml.libwebsockets.org/mailman/listinfo/libwebsockets 		 	   		  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Libwebsockets mailing list
>Libwebsockets at ml.libwebsockets.org
>http://ml.libwebsockets.org/mailman/listinfo/libwebsockets




More information about the Libwebsockets mailing list