[Libwebsockets] libwebsockets-1.4 HTTP server help with huge http chunks

Ash 20001 ash20001 at hotmail.com
Tue Jul 7 07:08:51 CEST 2015


I think I found the problem. In lws-plat-unix, where you go through the pending read list and fake POLLIN status, you use wsi->sock, which is incorrect, it should use the index that corresponds to the context->fds array index.
So this code seems to work for me, but I have no tested it thoroughly yet. (excuse the formatting)
#ifdef LWS_OPENSSL_SUPPORT        /*         * For all guys with buffered SSL read data already saved up, if they         * are not flowcontrolled, fake their POLLIN status so they'll get         * service to use up the buffered incoming data, even though their         * network socket may have nothing         */
        wsi = context->pending_read_list;        while (wsi) {          for (n = 0; n < context->fds_count; n++) {            if (context->fds[n].fd == wsi->sock)              { index = n; break;}          }
                wsi_next = wsi->pending_read_list_next;
                context->fds[index].revents |=                                                context->fds[index].events & POLLIN;                if (context->fds[index].revents & POLLIN) {                        /*                         * he's going to get serviced now, take him off the                         * list of guys with buffered SSL.  If he still has some                         * at the end of the service, he'll get put back on the                         * list then.                         */                        lws_ssl_remove_wsi_from_buffered_list(context, wsi);                }                wsi = wsi_next;        }#endif


From: andy.green at linaro.org
To: andy.green at linaro.org
Date: Mon, 6 Jul 2015 11:55:15 -0700
CC: libwebsockets at ml.libwebsockets.org
Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help with huge http chunks




Hello Roger,After spending more time debugging this issue, it looks like the data is stuck somewhere in the kernel. From what I see after the first 4096 packet is received, libwebsockets just goes into its normal 50 ms service loop with no indication that data is still pending to be read. During this time, the post request on the client is sitting waiting for the server to complete receiving all the data. But if I cancel the request (I used Postman for this) on the client side, the libwebsockets server side immediately gets the rest of the data.  However, I have to cancel within around 5 seconds (maybe some timeout somewhere) to get the data.
Does this make any sense to you in diagnosing the problem?
From: ash20001 at hotmail.com
To: andy.green at linaro.org
CC: libwebsockets at ml.libwebsockets.org
Subject: RE: [Libwebsockets] libwebsockets-1.4 HTTP server help with huge http chunks
Date: Thu, 2 Jul 2015 14:24:05 -0700




Hello Roger,Thank you for your investigation on this! I also found this same problem where the next wsi and the current wsi is the same and put in a fix myself for it but it is still not solving the issue. It is almost as if the SSL socket has no more read data in it even though it was sent. 
Here is an excerpt of the log when posting around 9K bytes of data to the server:
lwsts[13369]: known hdr 8lwsts[13369]: libwebsocket_parse sees parsing completelwsts[13369]: libwebsocket_ensure_user_space: 0x8971fc0 protocol 0x8063340lwsts[13369]: Method: POST request for '/'    post  = /    host: = 105.145.65.81:7681    connection: = keep-alive    origin: = chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop    http/1.1  = HTTP/1.1    accept: = */*    accept-encoding: = gzip, deflate    accept-language: = en-US,en;q=0.8    cache-control: = no-cache    cookie: = test=LWS_1435790394_190834_COOKIE    content-length: = 9438    content-type: = text/plain;charset=UTF-8    user-agent: = Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.130 Safari/537.36lwsts[13369]: libwebsocket_read: read_oklwsts[13369]: LWS_CALLBACK_HTTP_BODY: asdasdasdasasdasdasd... len 4096lwsts[13369]: libwebsocket_read: read_ok

As you can see the first 4k is received (with the body being 'asdasd...') and then it just spins in the main service loop.
> From: andy.green at linaro.org
> Date: Thu, 2 Jul 2015 20:41:13 +0100
> To: andy.green at linaro.org
> CC: libwebsockets at ml.libwebsockets.org
> Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help with huge	http chunks
> 
> Hi Ash,
> 
> I think I've found your problem, there is an attempt at a fix here:
> 
> https://github.com/warmcat/libwebsockets/commit/fc10940ec1c1e974a94eb5ed171f1aedaafef93e
> 
> Cheers,
> 
> Roger
> 
> 
> 
> On Thu, Jul 2, 2015 at 6:42 PM, Ash 20001 <andy.green at linaro.org> wrote:
> > I have confirmed this issue exists on all versions of websockets > 1.3 and
> > on the windows builds besides linux builds.
> >
> > This sounds like a bug in libwebsockets when using HTTP protocol over HTTPS
> > on the server side.
> > If anyone has any insight into how to fix the problem, it would be greatly
> > appreciated.
> >
> > ________________________________
> > From: andy.green at linaro.org
> > To: libwebsockets at ml.libwebsockets.org
> > Date: Tue, 30 Jun 2015 15:19:10 -0700
> >
> > Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help with huge
> > http chunks
> >
> > Sorry for the other emails, I have much easier reproducible steps for this
> > that maybe you guys can help me with:
> >
> > 1. Build 1.4 test-server binary.
> > 2. Run it with --ssl option.
> > 3. Use an HTTP client on another machine, like Advanced Rest Client to send
> > a POST data to https://<your server>:<your ssl port>. Start with under 4k
> > sized payloads and notice it works and server receives both BODY and BODY
> > COMPLETION.
> > 4. Change to higher than 4k POST data payload, and notice that the server
> > only receives the first 4k chunk and then gets stuck.
> >
> >
> >
> > ________________________________
> > From: andy.green at linaro.org
> > To: libwebsockets at ml.libwebsockets.org
> > Date: Tue, 30 Jun 2015 00:53:59 -0700
> > Subject: Re: [Libwebsockets] libwebsockets-1.4 HTTP server help with huge
> > http chunks
> >
> > One thing I forgot to mention: this only happens on HTTPS, not HTTP. HTTP
> > works fine. Any ideas why HTTPS would be doing this? It gets the first 4K
> > data just fine, just hangs after that and never gets the second or more 4K
> > data.
> >
> > ________________________________
> > From: andy.green at linaro.org
> > To: libwebsockets at ml.libwebsockets.org
> > Date: Mon, 29 Jun 2015 22:08:58 -0700
> > Subject: [Libwebsockets] libwebsockets-1.4 HTTP server help with huge http
> > chunks
> >
> > Hello,
> > I am trying to host an HTTP server (using the test-server.c sample code). I
> > have a client app that sends big chunks of HTTP data (around 10k-15k) to the
> > server.
> > The server doesn't seem to parse the body fragments well.
> >
> > By default libwebsockets delivers data in 4k chunks. This seems to work fine
> > for chunks less than 8k after adding libbwesockets_return_http_status with
> > HTTP_OK after each LWS_CALLBACK_HTTP_BODY message. In the case of less than
> > 8K and greater than 4K, I get two LWS_CALLBACK_HTTP_BODY message, the first
> > with 4k and then the remaining amount in the next message. Then I received
> > the LWS_CALLBACK_HTTP_BODY_COMPLETION message.
> >
> > But If I try something bigger than 8k, I never get the 3rd or higher chunk
> > in the BODY Message, and sometimes the whole client app errors out.  I
> > cannot find any sample out there to properly handle large HTTP chunks. I am
> > hoping maybe you guys might have something out there.
> >
> > Also, changing the rx_buffer_size in the http protocol seems to make no
> > difference.
> >
> > Thanks for any help!!
> >
> > _______________________________________________ Libwebsockets mailing list
> > Libwebsockets at ml.libwebsockets.org
> > http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
> >
> > _______________________________________________ Libwebsockets mailing list
> > Libwebsockets at ml.libwebsockets.org
> > http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
> >
> > _______________________________________________ Libwebsockets mailing list
> > Libwebsockets at ml.libwebsockets.org
> > http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
> >
> > _______________________________________________
> > Libwebsockets mailing list
> > Libwebsockets at ml.libwebsockets.org
> > http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
> >
> _______________________________________________
> Libwebsockets mailing list
> Libwebsockets at ml.libwebsockets.org
> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
 		 	   		   		 	   		  

_______________________________________________
Libwebsockets mailing list
Libwebsockets at ml.libwebsockets.org
http://ml.libwebsockets.org/mailman/listinfo/libwebsockets 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20150706/f93312b3/attachment-0001.html>


More information about the Libwebsockets mailing list