[Libwebsockets] libwebsockets HTTP mode lacks LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION

"Andy Green (林安廸)" andy at warmcat.com
Sun Nov 10 09:57:36 CET 2013


On 10/11/13 16:39, the mail apparently from Johan Lindh included:
> I take that to mean you've explicity disabled all IPv6 on your
> machines then. It's available and running by default on pretty much
> everything nowadays, desktop and server alike, Windows, OS/X and
> Linux. I'll certainly provide patches once done unless you beat me to
> it.

Some have it disabled in sysconfig others in the kernel config.  My ISP 
doesn't provide ipv6 routing.

> The benchmarks were not very scientific - server and client (ab -
> apachebench) both running on my MacBook Pro which has a dual core i5
> and runs OS/X 10.9. Take them with a large grain of salt. IMO,
> multithreading the server is mostly useful if a request blocks while
> the server generates something, which is not much of an issue serving
> static files in chunks. It's more likely to be the fact that pion
> supports HTTP/1.1 and can thus use Keep-Alive. TCP handshakes add up.
>
> On the subject of parsing.
>
> Your URI decoder look sane, but the path normalization looks wrong
> from what I can tell by looking at the patch. slash-dot-dot should
> back up a path level, not just be discarded, and path normalization

The code is different because it is a state machine.  It doesn't set 
slash_dot_dot until it has already handled the second dot and it only 
exists as a state to hoover up any more dots.

I tested it with wget and it seems to work.

> should stop at the query part (when '?' is seen). Also, make sure that
> you can't use percent encoding to avoid path normalization - I can't
> really tell from the patch if it does or not.

Yes the %xx stuff is shaken out first, then always goes through the same 
logic as the unencoded chars: that is why two kinds of state are tracked 
independently.

If anyone can find a bug obviously I'll be grateful for a fix.

About ? I don't think it's safe to just do that.  For example with the 
test app "/?/../../../../etc/passwd" will work (in a bad way) unless the 
userland code explicitly checks for ? as a delimiter.

You seem to be coming from a traditional webserver background where that 
would be a given to split out the args but it isn't for typical lws users.

I guess it needs extending to split it into a different "header" when ? 
is seen in the uri.

> You should store the request method and allow both GET and HEAD at a
> minimum. See http://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html for
> details. It should preferably also accept OPTIONS, POST, PUT and
> DELETE, even if the default implementation is just to spit back a "405
> Method Not Allowed".
>
> You should also keep the HTTP version parsed in the request, as the
> server needs to discern a HTTP/1.0 request from a HTTP/1.1 request.

Bear in mind this is originally intended for use to give a web UI in 
headless devices, we don't need to implement all that to be useful.

lws doesn't support keep-alive for http1.1 as it stands.

If you want to provide patches for these I'll add them, but we might be 
getting into needing a config option to stop the minimum build case 
bloating out with this stuff that's only useful in a completely 
different context.

-Andy

> /J
>
>
>
> On Sun, Nov 10, 2013 at 8:42 AM, "Andy Green (林安廸)" <andy at warmcat.com> wrote:
>> On 09/11/13 20:52, the mail apparently from Johan Lindh included:
>>
>>> I forgot to ask: Are you planning to implement IPv6 support?
>>>
>>> Currently, libwebsockets will only bind to the IPv4 address and most
>>> operating systems try IPv6 first, meaning at best address lookup is
>>> delayed (sometimes by several hundred ms) and at worst client
>>> connections simply fail.
>>
>>
>> I don't have an ipv6 setup here.
>>
>> I guess it's not that hard since it's only to do with socket-level
>> connection establishment / name resolution which is only a few dozen lines
>> of code.
>>
>> If you're interested in sending a patch I'm certainly interested to look at
>> it.
>>
>> -Andy
>>
>>
>>> /J
>>>
>>>
>>> On Sat, Nov 9, 2013 at 11:29 AM, Johan Lindh <johan at linkdata.se> wrote:
>>>>
>>>> Good work. Seems to work fine. I'm evaluating websocketpp alongside
>>>> with libwebsockets and pion. While pion is a fully featured HTTP
>>>> server and very fast, unfortunately it has no WebSocket support and no
>>>> apparent easy way to add it.
>>>>
>>>> In case you're interested, here's the current benchmarks, from slowest
>>>> to fastest, serving a static HTML file over 10 concurrent connections.
>>>> It's an apples-to-oranges comparison since pion is HTTP-only and
>>>> multithreaded to boot, but pion is a good benchmark for how fast you
>>>> can get serving HTTP with no shortcuts taken (as in, all headers fully
>>>> supported and HTTP/1.1).
>>>>
>>>> websocketpp 0.3.0-alpha4:
>>>> Time per request:       0.334 [ms] (mean, across all concurrent requests)
>>>> Transfer rate:          5658.60 [Kbytes/sec] received
>>>>
>>>> libwebsockets HEAD revision:
>>>> Time per request:       0.213 [ms] (mean, across all concurrent requests)
>>>> Transfer rate:          8508.20 [Kbytes/sec] received
>>>>
>>>> pion 5.0.4:
>>>> Time per request:       0.158 [ms] (mean, across all concurrent requests)
>>>> Transfer rate:          11745.94 [Kbytes/sec] received
>>>>
>>>> Considering that libwebsockets is a fraction of pion's size and has
>>>> WebSockets, it's looking very good indeed.
>>>>
>>>> You're missing a URL decoder though, so let me offer my own. It's
>>>> quite fast and also does path normalization so you can ease up on the
>>>> hardcore whitelisting in the test server. :-)
>>>>
>>>> /J
>>>>
>>>>
>>>> On Sat, Nov 9, 2013 at 5:08 AM, "Andy Green (林安廸)" <andy at warmcat.com>
>>>> wrote:
>>>>>
>>>>> On 09/11/13 08:28, the mail apparently from "Andy Green (林安廸)" included:
>>>>>
>>>>>> On 09/11/13 02:40, the mail apparently from Johan Lindh included:
>>>>>>
>>>>>>> Not having access to any client headers except the URI is unacceptable
>>>>>>
>>>>>>
>>>>>>
>>>>>> You don't have to accept it ^^
>>>>>>
>>>>>>> in LWS_CALLBACK_HTTP, and current HEAD in git does not issue
>>>>>>> WS_CALLBACK_FILTER_PROTOCOL_CONNECTION for HTTP connections.
>>>>>>
>>>>>>
>>>>>>
>>>>>> I can see it would be useful to do something, but we need to add an
>>>>>> enum
>>>>>> then, since that one is specifically when a websocket protocol
>>>>>> connection has been made.
>>>>>>
>>>>>>> Additionally, I strongly suggest adding "Accept:" and
>>>>>>> "If-Modified-Since:" to minilex.c - not being able to safely gzip or
>>>>>>> do basic cache handling cripples performance.
>>>>>>
>>>>>>
>>>>>>
>>>>>> More headers sounds reasonable.  At the moment its focus as you might
>>>>>> guess from the name is serving websocket protocol not http.  But the
>>>>>> http stuff could indeed cope with more things.
>>>>>>
>>>>>> Are there any other headers people are interested in processing and
>>>>>> having available inside libwebsockets?  They'd be handled for both pure
>>>>>> http and if they were present during the upgrade handshake.  It's
>>>>>> easier
>>>>>> to do them all at once since fiddling with minilex isn't simple (which
>>>>>> is why it got left to me I guess).
>>>>>
>>>>>
>>>>>
>>>>> I found myself with some free time thismorning.
>>>>>
>>>>> Additional headers supported in minilex + lws
>>>>>
>>>>>          "Accept:",
>>>>>          "If-Modified-Since:",
>>>>>          "Accept-Encoding:",
>>>>>          "Accept-Language:",
>>>>>          "Pragma:",
>>>>>          "Cache-Control:",
>>>>>          "Authorization:",
>>>>>          "Cookie:",
>>>>>          "Content-Type:",
>>>>>          "Date:",
>>>>>          "Range:",
>>>>>          "Referer:"
>>>>>
>>>>>
>>>>> http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=cc13c6f187f47fd31a377d05477f90c0fd7f7452
>>>>>
>>>>> introduce LWS_CALLBACK_FILTER_HTTP_CONNECTION
>>>>>
>>>>>
>>>>> http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=19895bcfd4e89f96100ab2dca041e965973f8e2b
>>>>>
>>>>> Fix ability to access headers in HTTP service
>>>>>
>>>>>
>>>>> http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=84fd949e05d011410dba2abe93dfd2fd7dff1589
>>>>>
>>>>> Add example code to test server for Cookies
>>>>>
>>>>>
>>>>> http://git.libwebsockets.org/cgi-bin/cgit/libwebsockets/commit/?id=6cd8880f23354ceeba75d641379c8f07407b0570
>>>>>
>>>>> -Andy
>>>>>
>>>>>
>>>>>
>>>>>> -Andy
>>>>>>
>>>>>>> Patch for lib/handshake.c attached.
>>>>>>>
>>>>>>> /Johan
>>>>>>>
>>>>>>
>>>>>> diff --git a/lib/handshake.c b/lib/handshake.c
>>>>>> index 3007426..00e3bc1 100644
>>>>>> --- a/lib/handshake.c
>>>>>> +++ b/lib/handshake.c
>>>>>> @@ -135,6 +135,11 @@ libwebsocket_read(struct libwebsocket_context
>>>>>> *context,
>>>>>>                 uri_ptr = lws_hdr_simple_ptr(wsi, WSI_TOKEN_GET_URI);
>>>>>>                 uri_len = lws_hdr_total_length(wsi, WSI_TOKEN_GET_URI);
>>>>>>
>>>>>> +            if (wsi->protocol->callback &&
>>>>>> wsi->protocol->callback(context, wsi,
>>>>>> +                    LWS_CALLBACK_FILTER_PROTOCOL_CONNECTION,
>>>>>> +                    wsi->user_space, uri_ptr, uri_len))
>>>>>> +                    goto bail_nuke_ah;
>>>>>> +
>>>>>>                 /* union transition */
>>>>>>                 memset(&wsi->u, 0, sizeof(wsi->u));
>>>>>>
>>>>>>
>>>>>
>>




More information about the Libwebsockets mailing list