[Libwebsockets] Socket close during some HTTP gets

Andy Green andy at warmcat.com
Thu Jan 19 04:22:18 CET 2017



On 01/19/2017 10:52 AM, Michael Behrns-Miller wrote:
> Hey Andy, thanks for being so helpful.
>
> I'm getting a similar result from the test client, see below.  No
> LWS_CALLBACK_COMPLETED_CLIENT_HTTP, had to hit Ctrl-C to exit.  I have
> pulled the latest v2.0-stable code.  Maybe I should switch up to v2.1?

It's best to use master if something needs changing.

>
> Thanks,
> Michael
>
>
> Here is the installed version of libwebsockets, it should be v2.0 or higher:
> 2.1.0
> m at case:~/development$ /usr/local/bin/libwebsockets-test-client
> https://www.google.com/#q=energy+news+today

That URL actually is chunked.



> [2017/01/18 21:47:43:2250] NOTICE: libwebsockets test client - license
> LGPL2.1+SLE
> [2017/01/18 21:47:43:2251] NOTICE: (C) Copyright 2010-2016 Andy Green
> <andy at warmcat.com>
> [2017/01/18 21:47:43:2251] NOTICE:  Using SSL
> [2017/01/18 21:47:43:2251] NOTICE:  Cert must validate correctly (use
> -s to allow selfsigned)
> [2017/01/18 21:47:43:2251] NOTICE:  Requiring peer cert hostname matches
> [2017/01/18 21:47:43:2251] NOTICE: Initial logging level 7
> [2017/01/18 21:47:43:2251] NOTICE: Libwebsockets version: 2.1.0
> m at case-v2.0.0-203-g109d66c
> [2017/01/18 21:47:43:2251] NOTICE: IPV6 not compiled in
> [2017/01/18 21:47:43:2251] NOTICE: libev support not compiled in
> [2017/01/18 21:47:43:2251] NOTICE: libuv support not compiled in
> [2017/01/18 21:47:43:2251] NOTICE:  Threads: 1 each 1024 fds
> [2017/01/18 21:47:43:2251] NOTICE:  mem: platform fd map:  8192 bytes
> [2017/01/18 21:47:43:2251] NOTICE:  Compiled with OpenSSL support
> [2017/01/18 21:47:43:2259] NOTICE: Creating Vhost 'default' port -1, 2
> protocols, IPv6 off
> [2017/01/18 21:47:43:2260] NOTICE:  mem: per-conn:          512 bytes
> + protocol rx buf
> [2017/01/18 21:47:43:2260] NOTICE: using https mode (non-ws)
> [2017/01/18 21:47:43:2260] NOTICE: http: connecting
> [2017/01/18 21:47:43:2260] NOTICE: lws_protocol_init
> [2017/01/18 21:47:43:5414] NOTICE:
> lws_client_interpret_server_handshake: client connection up
> [2017/01/18 21:47:43:5414] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5415] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5417] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5712] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5712] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5716] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5716] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5718] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5719] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5753] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5753] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5755] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5755] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5766] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5766] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5770] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5770] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:48:04:0004] NOTICE: wsi 0x556085b4dfb0: TIMEDOUT
> WAITING on 13 (did hdr 1, ah (nil), wl 0, pfd events 1)

Master client ends without the timeout, but he also doesn't get the 
content when it's chunked.

At any rate I can reproduce it, which makes it my problem.

I'll look at it later today.

-Andy

>
> On Wed, Jan 18, 2017 at 8:51 PM, Andy Green <andy at warmcat.com> wrote:
>>
>> On 01/19/2017 09:41 AM, Michael Behrns-Miller wrote:
>>> Thanks Andy.
>>>
>>> I did grab your latest code, my issue persists.  When i trace into the
>>> handling, I find that in service.c ln ~553 :spin_chunks label,
>>> wsi->chunked is 1.  I then have to catch my chunked http packets in
>>
>> It's possible the server you're talking to is just sending stuff with
>> Content-encoding: chunked
>>
>> CGIs tend to do it because they have no idea about the total length the
>> script will gente but want to send stuff as it becomes available.
>>
>>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ.  I'm afraid that I have not
>>> separated the wsi data of my websockets and http properly, and the wss
>>> chunked flag is set for my http traffic, maybe?
>> If you increase the logging to eg 63, you should see the http headers...
>> check and see if it is supposed to be chunked first... I assume it is.
>>
>> ws doesn't use HTTP chunked encoding... it's more likely your HTTP server
>> actually is sending chunked.
>>
>> What happens if you use the test client, eg
>>
>> $ libwebsockets-test-client http://warmcat.com
>>
>> will dump the HTML... does your server sending chunked kill it?  If so can I
>> try it from here?
>>
>> -Andy
>>
>>
>>> On Wed, Jan 18, 2017 at 8:34 PM, Andy Green <andy at warmcat.com> wrote:
>>>>
>>>> On 01/19/2017 06:25 AM, Michael Behrns-Miller wrote:
>>>>> Hello!
>>>>>
>>>>> I am working on interweaving an http get with a persistent wss client
>>>>> connection back to a server.  I am using one context and one protocol.
>>>>> lws always seems to go into "chunked' mode for the http get.  Some
>>>>
>>>> What do we mean by, 'chunked'... there is literally HTTP chunked mode,
>>>> which
>>>> has hex length headers before each chunk.  Or there is just "chunks" of
>>>> data
>>>> coming rather than the whole thing at once.
>>>>> sites work, and others drop the socket connection before finishing the
>>>>> http get.
>>>>>
>>>>> To get it working, there is a lot to get right, and i am probably
>>>>> getting some part of it wrong.
>>>>>
>>>>> My strategy was to use a standard wss client connection (that went
>>>>> fine).  Then for the http get, I create a second
>>>>> lws_client_connect_info and wsi for it, and attach it to the context
>>>>> with lws_client_connect_via_info().
>>>> Sounds good...
>>>>
>>>>> All my testing with local pages, big and small, seemed to work fine.
>>>>> Other tests (in this case a simple google query) gave this -d15 level
>>>>> debugging and failed to COMPLETE:
>>>>>
>>>>> [2017/01/18 16:42:09:3646] WARN: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>>>>> [2017/01/18 16:42:09:3646] WARN:
>>>>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ: received 365 bytes
>>>>> [2017/01/18 16:42:09:3646] WARN: HTTP_READ of 365 bytes begins:
>>>>> on(a){try{var b=docu
>>>>> [2017/01/18 16:42:09:3646] WARN: HTTP_READ of 365 bytes ends :
>>>>> /(^|\\?|&)ei=/.test(
>>>>> [2017/01/18 16:42:09:3646] WARN: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>>>>> [2017/01/18 16:42:09:3646] INFO: lws_close_free_wsi: real
>>>>> just_kill_connection: 0x5555557dfe90 (sockfd 7)
>>>>> [2017/01/18 16:42:09:3646] INFO: remove_wsi_socket_from_fds: removing
>>>>> same prot wsi 0x5555557dfe90
>>>>> [2017/01/18 16:42:09:3646] INFO: remove_wsi_socket_from_fds:
>>>>> wsi=0x5555557dfe90, sock=7, fds pos=2, end guy pos=3, endfd=0
>>>>> [2017/01/18 16:42:09:3647] INFO: lws_header_table_detach: wsi
>>>>> 0x5555557dfe90: ah (nil) (tsi=0, count = 0)
>>>>>
>>>>> Some of the http gets do not log any errors, but never finish.  In
>>>>> debugging the code, I usually find that the call is perceived as
>>>>> chunked for some reason, and I guess the final chunked marker is never
>>>>> hit.
>>>>>
>>>>> Sorry Andy, that's a small bit of the picture I know.  Any advice?
>>>>> Even if it is just something to try.  Appreciate it.
>>>> Make sure you're using latest master lws first I think.  I fixed some
>>>> bugs
>>>> around client HTTP last week.
>>>>
>>>> There's nothing wrong with the overall approach you can have any mixture
>>>> of
>>>> client connections as much as you like.
>>>>
>>>> -Andy
>>>>>
>>>>> Michael
>>>>> _______________________________________________
>>>>> Libwebsockets mailing list
>>>>> Libwebsockets at ml.libwebsockets.org
>>>>> https://libwebsockets.org/mailman/listinfo/libwebsockets
>>>>




More information about the Libwebsockets mailing list