[Libwebsockets] client chunk/unchunk

Andy Green andy at warmcat.com
Mon Mar 6 20:38:28 CET 2017



On March 7, 2017 3:28:00 AM GMT+08:00, Joel Winarske <joel.winarske at gmail.com> wrote:
>I sorted it out.  I am now coalescing the http client header/reads into
>a
>linked list.  I start draining it with a priming call to
>lws_callback_on_writable(), which then is called from
>LWS_CALLBACK_HTTP_WRITEABLE until it's empty, at which point I attempt
>to
>re-use the connection.
>
>It would be cool If there was a way to avoid the memcpy for perf
>reasons,
>but for now it's working.

The data we got in that case has the chunking headers sitting there inline with the payload, and someone has to conceal them... lws could either

 - let you ask for one chunk at a time and give an indication if more left so you can ask for another writable callback... performance would reflect the chunk sizes the remote guy sent plus or minus tcp fragmentation of it

 - do the memcpys itself to coalesce before giving it to you as one chunk (needs some care on lws part in the case the rx packet we got ends with part of a chunk header)

If you are not passing the data on to another fd, you likely don't care about getting it in pieces, the memcpys faking it being one chunk are then wasted overhead if you anyway memcpy it somewhere else.

-Andy

>Thanks,
>Joel
>
>On Mon, Mar 6, 2017 at 8:15 AM, Joel Winarske <joel.winarske at gmail.com>
>wrote:
>
>> The chunk/unchunk is fine.  In my case the content-length was not
>equal to
>> the chunk/unchunk read sizes.  This threw me off.  My problem lies in
>> trying to re-write these chunked/unchunked reads to the parent wsi.
>>
>> At the end of each read I call:
>>     lws_callback_on_writable(lws_get_parent(wsi))
>>
>> Below is the output pattern of using this method.  I see the parent
>> writable callback happen every other time.
>>
>> Do I need to coalesce the read data, or should I see the writable
>callback
>> happen after each read, or do I ignore using
>lws_callback_on_writable() and
>> just write it, since it's waiting for the response anyway?
>>
>>
>> [2017/03/06 08:00:00:7171] INFO: http: (0301F760)
>LWS_CALLBACK_ESTABLISHED_CLIENT_HTTP:
>> 200
>> [2017/03/06 08:00:00:7181] INFO: lws_header_table_detach: wsi
>0301F760: ah
>> 00EE43B8 (tsi=0, count = 2)
>> [2017/03/06 08:00:00:7191] INFO: lws_header_table_detach: wsi
>0301F760: ah
>> 00EE43B8 (tsi=0, count = 1)
>> [2017/03/06 08:00:00:7201] NOTICE:
>lws_client_interpret_server_handshake:
>> client connection up
>> [2017/03/06 08:00:00:7201] INFO: ssl buffered read
>> [2017/03/06 08:00:00:7211] INFO: chunk 32768
>> [2017/03/06 08:00:00:7221] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 701
>> [2017/03/06 08:00:00:7241] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7261] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7261] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7271] INFO: http: (0301F450)
>> LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>> [2017/03/06 08:00:00:7271] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7281] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7281] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7291] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7291] INFO: http: (0301F450)
>> LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>> [2017/03/06 08:00:00:7301] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7301] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7311] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7321] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7331] INFO: http: (0301F450)
>> LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>> [2017/03/06 08:00:00:7341] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7371] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7371] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7381] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7391] INFO: http: (0301F450)
>> LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>> [2017/03/06 08:00:00:7391] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7401] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7401] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7401] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7411] INFO: http: (0301F450)
>> LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>> [2017/03/06 08:00:00:7411] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7421] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7431] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7431] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7431] INFO: http: (0301F450)
>> LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>> [2017/03/06 08:00:00:7441] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7441] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7451] INFO: http: (0301F760)
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>> [2017/03/06 08:00:00:7451] INFO: lws_read: read_ok, used 0
>> [2017/03/06 08:00:00:7461] NOTICE: lws_server_socket_service: wsi
>0301F450
>> read 0
>> [2017/03/06 08:00:00:7501] INFO: lws_server_socket_service: read 0
>len
>> [2017/03/06 08:00:00:7511] ERR: 169.254.225.161 - -
>[06/Mar/2017:08:00:00
>> -0800] "POST /rest/send http/1.1" 200 8976 Mozilla/5.0 (Windows NT
>10.0;
>> WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87
>> Safari/537.36
>> [2017/03/06 08:00:00:7521] INFO: lws_close_free_wsi: real
>> just_kill_connection: 0301F760 (sockfd 560)
>> [2017/03/06 08:00:00:7531] INFO: remove_wsi_socket_from_fds: removing
>same
>> prot wsi 0301F760
>> [2017/03/06 08:00:00:7531] INFO: lws_close_free_wsi: shutting down
>> connection: 0301F450 (sock 500, state 3)
>> [2017/03/06 08:00:00:7541] INFO: lws_close_free_wsi: real
>> just_kill_connection: 0301F450 (sockfd 500)
>> [2017/03/06 08:00:00:7541] INFO: remove_wsi_socket_from_fds: removing
>same
>> prot wsi 0301F450
>> [2017/03/06 08:00:00:7551] INFO: ah det due to close
>> [2017/03/06 08:00:00:7551] INFO: lws_header_table_detach: wsi
>0301F450: ah
>> 00EE35A0 (tsi=0, count = 1)
>> [2017/03/06 08:00:00:7561] INFO: lws_header_table_detach: wsi
>0301F450: ah
>> 00EE35A0 (tsi=0, count = 0)
>> [2017/03/06 08:00:00:7571] INFO: ah det due to close
>> [2017/03/06 08:00:00:7571] INFO: lws_header_table_detach: wsi
>0301F760: ah
>> 00000000 (tsi=0, count = 0)
>> [2017/03/06 08:00:00:7581] INFO: lws_header_table_detach: wsi
>0301F760: ah
>> 00000000 (tsi=0, count = 0)
>>
>>
>>
>>
>>
>> On Sun, Mar 5, 2017 at 8:43 PM Andy Green <andy at warmcat.com> wrote:
>>
>>>
>>>
>>> On 03/05/2017 11:55 PM, Joel Winarske wrote:
>>> > Hi Andy,
>>> >
>>> > I'm seeing a problem with the client.  lws_http_client_read()
>returns
>>> > a value in both RECEIVE_CLIENT_HTTP callbacks.  I would expect it
>>> > would only return a value for chunked given the size.  See
>anything
>>> > suspicious here?
>>> >
>>> > [2017/03/05 07:51:40:9226] NOTICE: lws_ssl_client_connect2:
>>> > SSL_connect says 1
>>> > [2017/03/05 07:51:41:4546] INFO: lws_ensure_user_space: 031651B0
>>> > protocol 02E25628
>>> > [2017/03/05 07:51:41:4556] INFO: lws_ensure_user_space: 031651B0
>>> > protocol pss 11308, user_space=03173C10
>>> > [2017/03/05 07:51:41:4566] NOTICE:
>>> > lws_client_interpret_server_handshake: incoming content length
>9506
>>> >     connection: keep-alive
>>> >     http/1.1  200 OK
>>> >     pragma: no-cache
>>> >     cache-control: no-cache
>>> >     content-length: 9506
>>> >     content-type: application/json; charset=utf-8
>>> >     date: Sun, 05 Mar 2017 15:51:38 GMT
>>> >     expires: -1
>>> >     server: Microsoft-IIS/8.5
>>> > [2017/03/05 07:51:41:4626] INFO: lws_header_table_detach: wsi
>>> > 031651B0: ah 00E53258 (tsi=0, count = 2)
>>> > [2017/03/05 07:51:41:4636] INFO: lws_header_table_detach: wsi
>>> > 031651B0: ah 00E53258 (tsi=0, count = 1)
>>> > [2017/03/05 07:51:41:4636] NOTICE:
>>> > lws_client_interpret_server_handshake: client connection up
>>> > [2017/03/05 07:51:41:4636] INFO: ssl buffered read
>>> > [2017/03/05 07:51:41:4646] INFO: http: (031651B0)
>>> > LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 4768
>>> > [2017/03/05 07:51:41:4646] INFO: http: (031651B0)
>>> > LWS_CALLBACK_RECEIVE_CLIENT_HTTP (unchunked) 4768
>>>
>>> How about only taking data from
>LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ,
>>> for both chunked and unchunked?
>>>
>>> -Andy
>>>
>>> > [2017/03/05 07:51:41:4656] INFO: lws_close_free_wsi: shutting down
>SSL
>>> > connection: 031651B0 (ssl 0317AD78, sock 560, state 6)
>>> > [2017/03/05 07:51:41:4666] INFO: lws_close_free_wsi: real
>>> > just_kill_connection: 031651B0 (sockfd 560)
>>> > [2017/03/05 07:51:41:4676] INFO: remove_wsi_socket_from_fds:
>removing
>>> > same prot wsi 031651B0
>>> > [2017/03/05 07:51:41:4676] INFO: ah det due to close
>>> > [2017/03/05 07:51:41:4686] INFO: lws_header_table_detach: wsi
>>> > 031651B0: ah 00000000 (tsi=0, count = 1)
>>> > [2017/03/05 07:51:41:4686] INFO: lws_header_table_detach: wsi
>>> > 031651B0: ah 00000000 (tsi=0, count = 1)
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Libwebsockets mailing list
>>> > Libwebsockets at ml.libwebsockets.org
>>> > https://libwebsockets.org/mailman/listinfo/libwebsockets
>>>
>>>



More information about the Libwebsockets mailing list