[Libwebsockets] Socket close during some HTTP gets

Michael Behrns-Miller moodboom at gmail.com
Thu Jan 19 04:19:38 CET 2017


Note: Apparently I had just switched to the v2.1 branch before
building and running that test.  I did a fresh pull.

Also of note: my code didn't need any changes for 2.0 to 2.1.  Looks
like I have made the 2.1 upgrade with zero effort, cool.  :-)

On Wed, Jan 18, 2017 at 9:52 PM, Michael Behrns-Miller
<moodboom at gmail.com> wrote:
> Hey Andy, thanks for being so helpful.
>
> I'm getting a similar result from the test client, see below.  No
> LWS_CALLBACK_COMPLETED_CLIENT_HTTP, had to hit Ctrl-C to exit.  I have
> pulled the latest v2.0-stable code.  Maybe I should switch up to v2.1?
>
> Thanks,
> Michael
>
>
> Here is the installed version of libwebsockets, it should be v2.0 or higher:
> 2.1.0
> m at case:~/development$ /usr/local/bin/libwebsockets-test-client
> https://www.google.com/#q=energy+news+today
> [2017/01/18 21:47:43:2250] NOTICE: libwebsockets test client - license
> LGPL2.1+SLE
> [2017/01/18 21:47:43:2251] NOTICE: (C) Copyright 2010-2016 Andy Green
> <andy at warmcat.com>
> [2017/01/18 21:47:43:2251] NOTICE:  Using SSL
> [2017/01/18 21:47:43:2251] NOTICE:  Cert must validate correctly (use
> -s to allow selfsigned)
> [2017/01/18 21:47:43:2251] NOTICE:  Requiring peer cert hostname matches
> [2017/01/18 21:47:43:2251] NOTICE: Initial logging level 7
> [2017/01/18 21:47:43:2251] NOTICE: Libwebsockets version: 2.1.0
> m at case-v2.0.0-203-g109d66c
> [2017/01/18 21:47:43:2251] NOTICE: IPV6 not compiled in
> [2017/01/18 21:47:43:2251] NOTICE: libev support not compiled in
> [2017/01/18 21:47:43:2251] NOTICE: libuv support not compiled in
> [2017/01/18 21:47:43:2251] NOTICE:  Threads: 1 each 1024 fds
> [2017/01/18 21:47:43:2251] NOTICE:  mem: platform fd map:  8192 bytes
> [2017/01/18 21:47:43:2251] NOTICE:  Compiled with OpenSSL support
> [2017/01/18 21:47:43:2259] NOTICE: Creating Vhost 'default' port -1, 2
> protocols, IPv6 off
> [2017/01/18 21:47:43:2260] NOTICE:  mem: per-conn:          512 bytes
> + protocol rx buf
> [2017/01/18 21:47:43:2260] NOTICE: using https mode (non-ws)
> [2017/01/18 21:47:43:2260] NOTICE: http: connecting
> [2017/01/18 21:47:43:2260] NOTICE: lws_protocol_init
> [2017/01/18 21:47:43:5414] NOTICE:
> lws_client_interpret_server_handshake: client connection up
> [2017/01/18 21:47:43:5414] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5415] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5417] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5712] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5712] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5716] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5716] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5718] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5719] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5753] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5753] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5755] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5755] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5766] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5766] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5770] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5770] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
> [2017/01/18 21:48:04:0004] NOTICE: wsi 0x556085b4dfb0: TIMEDOUT
> WAITING on 13 (did hdr 1, ah (nil), wl 0, pfd events 1)
>
> On Wed, Jan 18, 2017 at 8:51 PM, Andy Green <andy at warmcat.com> wrote:
>>
>>
>> On 01/19/2017 09:41 AM, Michael Behrns-Miller wrote:
>>>
>>> Thanks Andy.
>>>
>>> I did grab your latest code, my issue persists.  When i trace into the
>>> handling, I find that in service.c ln ~553 :spin_chunks label,
>>> wsi->chunked is 1.  I then have to catch my chunked http packets in
>>
>>
>> It's possible the server you're talking to is just sending stuff with
>> Content-encoding: chunked
>>
>> CGIs tend to do it because they have no idea about the total length the
>> script will gente but want to send stuff as it becomes available.
>>
>>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ.  I'm afraid that I have not
>>> separated the wsi data of my websockets and http properly, and the wss
>>> chunked flag is set for my http traffic, maybe?
>>
>> If you increase the logging to eg 63, you should see the http headers...
>> check and see if it is supposed to be chunked first... I assume it is.
>>
>> ws doesn't use HTTP chunked encoding... it's more likely your HTTP server
>> actually is sending chunked.
>>
>> What happens if you use the test client, eg
>>
>> $ libwebsockets-test-client http://warmcat.com
>>
>> will dump the HTML... does your server sending chunked kill it?  If so can I
>> try it from here?
>>
>> -Andy
>>
>>
>>>
>>> On Wed, Jan 18, 2017 at 8:34 PM, Andy Green <andy at warmcat.com> wrote:
>>>>
>>>>
>>>> On 01/19/2017 06:25 AM, Michael Behrns-Miller wrote:
>>>>>
>>>>> Hello!
>>>>>
>>>>> I am working on interweaving an http get with a persistent wss client
>>>>> connection back to a server.  I am using one context and one protocol.
>>>>> lws always seems to go into "chunked' mode for the http get.  Some
>>>>
>>>>
>>>> What do we mean by, 'chunked'... there is literally HTTP chunked mode,
>>>> which
>>>> has hex length headers before each chunk.  Or there is just "chunks" of
>>>> data
>>>> coming rather than the whole thing at once.
>>>>>
>>>>> sites work, and others drop the socket connection before finishing the
>>>>> http get.
>>>>>
>>>>> To get it working, there is a lot to get right, and i am probably
>>>>> getting some part of it wrong.
>>>>>
>>>>> My strategy was to use a standard wss client connection (that went
>>>>> fine).  Then for the http get, I create a second
>>>>> lws_client_connect_info and wsi for it, and attach it to the context
>>>>> with lws_client_connect_via_info().
>>>>
>>>> Sounds good...
>>>>
>>>>> All my testing with local pages, big and small, seemed to work fine.
>>>>> Other tests (in this case a simple google query) gave this -d15 level
>>>>> debugging and failed to COMPLETE:
>>>>>
>>>>> [2017/01/18 16:42:09:3646] WARN: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>>>>> [2017/01/18 16:42:09:3646] WARN:
>>>>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ: received 365 bytes
>>>>> [2017/01/18 16:42:09:3646] WARN: HTTP_READ of 365 bytes begins:
>>>>> on(a){try{var b=docu
>>>>> [2017/01/18 16:42:09:3646] WARN: HTTP_READ of 365 bytes ends :
>>>>> /(^|\\?|&)ei=/.test(
>>>>> [2017/01/18 16:42:09:3646] WARN: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>>>>> [2017/01/18 16:42:09:3646] INFO: lws_close_free_wsi: real
>>>>> just_kill_connection: 0x5555557dfe90 (sockfd 7)
>>>>> [2017/01/18 16:42:09:3646] INFO: remove_wsi_socket_from_fds: removing
>>>>> same prot wsi 0x5555557dfe90
>>>>> [2017/01/18 16:42:09:3646] INFO: remove_wsi_socket_from_fds:
>>>>> wsi=0x5555557dfe90, sock=7, fds pos=2, end guy pos=3, endfd=0
>>>>> [2017/01/18 16:42:09:3647] INFO: lws_header_table_detach: wsi
>>>>> 0x5555557dfe90: ah (nil) (tsi=0, count = 0)
>>>>>
>>>>> Some of the http gets do not log any errors, but never finish.  In
>>>>> debugging the code, I usually find that the call is perceived as
>>>>> chunked for some reason, and I guess the final chunked marker is never
>>>>> hit.
>>>>>
>>>>> Sorry Andy, that's a small bit of the picture I know.  Any advice?
>>>>> Even if it is just something to try.  Appreciate it.
>>>>
>>>> Make sure you're using latest master lws first I think.  I fixed some
>>>> bugs
>>>> around client HTTP last week.
>>>>
>>>> There's nothing wrong with the overall approach you can have any mixture
>>>> of
>>>> client connections as much as you like.
>>>>
>>>> -Andy
>>>>>
>>>>>
>>>>> Michael
>>>>> _______________________________________________
>>>>> Libwebsockets mailing list
>>>>> Libwebsockets at ml.libwebsockets.org
>>>>> https://libwebsockets.org/mailman/listinfo/libwebsockets
>>>>
>>>>
>>



More information about the Libwebsockets mailing list