[Libwebsockets] Socket close during some HTTP gets

Michael Behrns-Miller moodboom at gmail.com
Thu Jan 19 04:47:47 CET 2017


Thanks Andy.

FYI If I put in breakpoints or delays it would often be enough to
trigger successful completion.  Let me know if I can help in any way.

On Wed, Jan 18, 2017 at 10:22 PM, Andy Green <andy at warmcat.com> wrote:
>
>
> On 01/19/2017 10:52 AM, Michael Behrns-Miller wrote:
>>
>> Hey Andy, thanks for being so helpful.
>>
>> I'm getting a similar result from the test client, see below.  No
>> LWS_CALLBACK_COMPLETED_CLIENT_HTTP, had to hit Ctrl-C to exit.  I have
>> pulled the latest v2.0-stable code.  Maybe I should switch up to v2.1?
>
>
> It's best to use master if something needs changing.
>
>>
>> Thanks,
>> Michael
>>
>>
>> Here is the installed version of libwebsockets, it should be v2.0 or
>> higher:
>> 2.1.0
>> m at case:~/development$ /usr/local/bin/libwebsockets-test-client
>> https://www.google.com/#q=energy+news+today
>
>
> That URL actually is chunked.
>
>
>
>
>> [2017/01/18 21:47:43:2250] NOTICE: libwebsockets test client - license
>> LGPL2.1+SLE
>> [2017/01/18 21:47:43:2251] NOTICE: (C) Copyright 2010-2016 Andy Green
>> <andy at warmcat.com>
>> [2017/01/18 21:47:43:2251] NOTICE:  Using SSL
>> [2017/01/18 21:47:43:2251] NOTICE:  Cert must validate correctly (use
>> -s to allow selfsigned)
>> [2017/01/18 21:47:43:2251] NOTICE:  Requiring peer cert hostname matches
>> [2017/01/18 21:47:43:2251] NOTICE: Initial logging level 7
>> [2017/01/18 21:47:43:2251] NOTICE: Libwebsockets version: 2.1.0
>> m at case-v2.0.0-203-g109d66c
>> [2017/01/18 21:47:43:2251] NOTICE: IPV6 not compiled in
>> [2017/01/18 21:47:43:2251] NOTICE: libev support not compiled in
>> [2017/01/18 21:47:43:2251] NOTICE: libuv support not compiled in
>> [2017/01/18 21:47:43:2251] NOTICE:  Threads: 1 each 1024 fds
>> [2017/01/18 21:47:43:2251] NOTICE:  mem: platform fd map:  8192 bytes
>> [2017/01/18 21:47:43:2251] NOTICE:  Compiled with OpenSSL support
>> [2017/01/18 21:47:43:2259] NOTICE: Creating Vhost 'default' port -1, 2
>> protocols, IPv6 off
>> [2017/01/18 21:47:43:2260] NOTICE:  mem: per-conn:          512 bytes
>> + protocol rx buf
>> [2017/01/18 21:47:43:2260] NOTICE: using https mode (non-ws)
>> [2017/01/18 21:47:43:2260] NOTICE: http: connecting
>> [2017/01/18 21:47:43:2260] NOTICE: lws_protocol_init
>> [2017/01/18 21:47:43:5414] NOTICE:
>> lws_client_interpret_server_handshake: client connection up
>> [2017/01/18 21:47:43:5414] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5415] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5416] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5417] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5418] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5712] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5712] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5713] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5714] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5716] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5716] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5718] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5719] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5743] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5753] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5753] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5755] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5755] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5766] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5766] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5770] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5770] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:47:43:5788] NOTICE: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>> [2017/01/18 21:48:04:0004] NOTICE: wsi 0x556085b4dfb0: TIMEDOUT
>> WAITING on 13 (did hdr 1, ah (nil), wl 0, pfd events 1)
>
>
> Master client ends without the timeout, but he also doesn't get the content
> when it's chunked.
>
> At any rate I can reproduce it, which makes it my problem.
>
> I'll look at it later today.
>
>
> -Andy
>
>>
>> On Wed, Jan 18, 2017 at 8:51 PM, Andy Green <andy at warmcat.com> wrote:
>>>
>>>
>>> On 01/19/2017 09:41 AM, Michael Behrns-Miller wrote:
>>>>
>>>> Thanks Andy.
>>>>
>>>> I did grab your latest code, my issue persists.  When i trace into the
>>>> handling, I find that in service.c ln ~553 :spin_chunks label,
>>>> wsi->chunked is 1.  I then have to catch my chunked http packets in
>>>
>>>
>>> It's possible the server you're talking to is just sending stuff with
>>> Content-encoding: chunked
>>>
>>> CGIs tend to do it because they have no idea about the total length the
>>> script will gente but want to send stuff as it becomes available.
>>>
>>>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ.  I'm afraid that I have not
>>>> separated the wsi data of my websockets and http properly, and the wss
>>>> chunked flag is set for my http traffic, maybe?
>>>
>>> If you increase the logging to eg 63, you should see the http headers...
>>> check and see if it is supposed to be chunked first... I assume it is.
>>>
>>> ws doesn't use HTTP chunked encoding... it's more likely your HTTP server
>>> actually is sending chunked.
>>>
>>> What happens if you use the test client, eg
>>>
>>> $ libwebsockets-test-client http://warmcat.com
>>>
>>> will dump the HTML... does your server sending chunked kill it?  If so
>>> can I
>>> try it from here?
>>>
>>> -Andy
>>>
>>>
>>>> On Wed, Jan 18, 2017 at 8:34 PM, Andy Green <andy at warmcat.com> wrote:
>>>>>
>>>>>
>>>>> On 01/19/2017 06:25 AM, Michael Behrns-Miller wrote:
>>>>>>
>>>>>> Hello!
>>>>>>
>>>>>> I am working on interweaving an http get with a persistent wss client
>>>>>> connection back to a server.  I am using one context and one protocol.
>>>>>> lws always seems to go into "chunked' mode for the http get.  Some
>>>>>
>>>>>
>>>>> What do we mean by, 'chunked'... there is literally HTTP chunked mode,
>>>>> which
>>>>> has hex length headers before each chunk.  Or there is just "chunks" of
>>>>> data
>>>>> coming rather than the whole thing at once.
>>>>>>
>>>>>> sites work, and others drop the socket connection before finishing the
>>>>>> http get.
>>>>>>
>>>>>> To get it working, there is a lot to get right, and i am probably
>>>>>> getting some part of it wrong.
>>>>>>
>>>>>> My strategy was to use a standard wss client connection (that went
>>>>>> fine).  Then for the http get, I create a second
>>>>>> lws_client_connect_info and wsi for it, and attach it to the context
>>>>>> with lws_client_connect_via_info().
>>>>>
>>>>> Sounds good...
>>>>>
>>>>>> All my testing with local pages, big and small, seemed to work fine.
>>>>>> Other tests (in this case a simple google query) gave this -d15 level
>>>>>> debugging and failed to COMPLETE:
>>>>>>
>>>>>> [2017/01/18 16:42:09:3646] WARN: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>>>>>> [2017/01/18 16:42:09:3646] WARN:
>>>>>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ: received 365 bytes
>>>>>> [2017/01/18 16:42:09:3646] WARN: HTTP_READ of 365 bytes begins:
>>>>>> on(a){try{var b=docu
>>>>>> [2017/01/18 16:42:09:3646] WARN: HTTP_READ of 365 bytes ends :
>>>>>> /(^|\\?|&)ei=/.test(
>>>>>> [2017/01/18 16:42:09:3646] WARN: LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>>>>>> [2017/01/18 16:42:09:3646] INFO: lws_close_free_wsi: real
>>>>>> just_kill_connection: 0x5555557dfe90 (sockfd 7)
>>>>>> [2017/01/18 16:42:09:3646] INFO: remove_wsi_socket_from_fds: removing
>>>>>> same prot wsi 0x5555557dfe90
>>>>>> [2017/01/18 16:42:09:3646] INFO: remove_wsi_socket_from_fds:
>>>>>> wsi=0x5555557dfe90, sock=7, fds pos=2, end guy pos=3, endfd=0
>>>>>> [2017/01/18 16:42:09:3647] INFO: lws_header_table_detach: wsi
>>>>>> 0x5555557dfe90: ah (nil) (tsi=0, count = 0)
>>>>>>
>>>>>> Some of the http gets do not log any errors, but never finish.  In
>>>>>> debugging the code, I usually find that the call is perceived as
>>>>>> chunked for some reason, and I guess the final chunked marker is never
>>>>>> hit.
>>>>>>
>>>>>> Sorry Andy, that's a small bit of the picture I know.  Any advice?
>>>>>> Even if it is just something to try.  Appreciate it.
>>>>>
>>>>> Make sure you're using latest master lws first I think.  I fixed some
>>>>> bugs
>>>>> around client HTTP last week.
>>>>>
>>>>> There's nothing wrong with the overall approach you can have any
>>>>> mixture
>>>>> of
>>>>> client connections as much as you like.
>>>>>
>>>>> -Andy
>>>>>>
>>>>>>
>>>>>> Michael
>>>>>> _______________________________________________
>>>>>> Libwebsockets mailing list
>>>>>> Libwebsockets at ml.libwebsockets.org
>>>>>> https://libwebsockets.org/mailman/listinfo/libwebsockets
>>>>>
>>>>>
>



More information about the Libwebsockets mailing list