[Libwebsockets] client chunk/unchunk

Andy Green andy at warmcat.com
Tue Mar 7 01:53:31 CET 2017



On 03/07/2017 04:18 AM, Joel Winarske wrote:
> Yes this is my situation.  So then if the header contains 
> "transfer-encoding: chunked", I would want the t-e headers preserved 
> in the read chunks.  If "transfer-encoding: chunked" was not present 
> in the header, the t-e headers would be stripped.  Are there any other 
> corner cases I'm missing?

I'm not sure what you mean by the last bit, if there is no t-e header 
then there is no t-e overhead, content-length is required and we know 
the end of the transaction from counting the number of payload bytes 
matched that.  If there was t-e chunked header, we deal with that until 
now by concealing the chunk headers spread throughout the payload.  If 
you will process or change the payload before passing it on / using it, 
that is what you need, since the chunk headers can come at any time 
including in the middle of variable names, etc, you have to get rid of 
them then.

If you are passing the incoming payload on over another http link (like 
proxying) unchanged, and you are forced to use t-e chunked there too, 
because previously chunked payload started coming and there was no 
content-length hint (you could buffer the whole thing until you know the 
payload length, but for anything nontrivial in length it's too expensive 
and isn't The Lws Way).

There are two approaches to come at it, lws could add a mode where it 
passes through the original chunking headers as payload, and you pass 
everything on.  However that leaves no room for you to modify the 
content, since the chunking headers won't match.

Therefore I think you should solve it the second way, which is continue 
to strip the old chunking headers, and add on your own chunking headers 
when you send the possibly-modified payload-only chunks back out.  This 
seems more correct to me since "Transfer Encoding" applies to the 
transfer from the remote server to your client connection, the outgoing 
transfer may choose a different encoding or buffer stuff as well as 
modify it so the old chunk headers no longer apply.

The format is very simple

https://en.wikipedia.org/wiki/Chunked_transfer_encoding

Lws could help by adding one of the two options I mentioned before, a 
single READ callback mode, or memmove() the buffer inside lws to elide 
the chunking headers so there is only one payload-only chunk left 
linearly in the buffer.  But I think now is not a good time to add this 
since I will make a release shortly and this potentially affects a lot 
of different things.

So a) add your own t-e: chunked header and b) add your own chunking 
headers on what you have already.


>
> https://www.google.com has "transfer-encoding: chunked".

Yeah anything produced by a script (ie, all nontrivial cgi) is likely in 
t-e chunked.  Because the script generates the headers and doesn't know 
generally how much it will produce exactly, so it cannot issue 
content-length at the start.  Google in particular don't want to wait to 
finish rendering your custom dynamic search homepage so they know the 
content-length before they start to send it.

-Andy

>
> On Mon, Mar 6, 2017 at 11:46 AM, Andy Green <andy at warmcat.com 
> <mailto:andy at warmcat.com>> wrote:
>
>
>
>     On March 7, 2017 3:37:58 AM GMT+08:00, Joel Winarske
>     <joel.winarske at gmail.com <mailto:joel.winarske at gmail.com>> wrote:
>     >Okay, so this makes a good deal more sense now :)
>     >
>     >In the case of forwarding the coalesced chunks, do I need to append
>     >anything for the chunking transfer encoding inline headers, or does
>     >that
>     >happen magically?
>
>     Any incoming chunked transfer-encoding headers are stripped by lws
>     by the time you receive the data... lws removes the t-e headers
>     and just gives you seamless payload.  So you are not required to
>     pass it on also in chunked t-e.
>
>     If you want to do that (because you don't know the final
>     content-length maybe) you need a new option somewhere in lws to
>     ask it to pass through the t-e headers.  Is that your situation?
>
>     To answer your question yes you will have to set the t-e header
>     yourself if you want to send chunked.
>
>     -Andy
>
>     >On Mon, Mar 6, 2017 at 11:05 AM, Andy Green <andy at warmcat.com
>     <mailto:andy at warmcat.com>> wrote:
>     >
>     >>
>     >>
>     >> On March 7, 2017 12:15:53 AM GMT+08:00, Joel Winarske <
>     >> joel.winarske at gmail.com <mailto:joel.winarske at gmail.com>> wrote:
>     >> >The chunk/unchunk is fine.  In my case the content-length was not
>     >equal
>     >> >to
>     >> >the chunk/unchunk read sizes.  This threw me off.  My problem lies
>     >in
>     >> >trying to re-write these chunked/unchunked reads to the parent
>     wsi.
>     >> >
>     >> >At the end of each read I call:
>     >> >    lws_callback_on_writable(lws_get_parent(wsi))
>     >> >
>     >> >Below is the output pattern of using this method.  I see the
>     parent
>     >> >writable callback happen every other time.
>     >> >
>     >> >Do I need to coalesce the read data, or should I see the writable
>     >> >callback
>     >> >happen after each read, or do I ignore using
>     >lws_callback_on_writable()
>     >> >and
>     >> >just write it, since it's waiting for the response anyway?
>     >>
>     >> Maybe it's better to answer by explaining what's happening overall
>     >>
>     >>  - LWS_CALLBACK_RECEIVE_CLIENT_HTTP comes... it means "hey the
>     remote
>     >> server sent us some rx data, I didn't look at it yet"
>     >>
>     >>  - you don't have to deal with it (by lws_http_client_read()) right
>     >there
>     >> in the callback.  It's just an indication something can be read.
>     >>
>     >>  - instead you can set a flag and ask for a callback on
>     writeable on
>     >your
>     >> outbound wsi.
>     >>
>     >>  - when the outbound wsi is writeable, and the flag set, you
>     can call
>     >> lws_http_client_read() from there
>     >>
>     >>  - with unchunked, you should get one
>     >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ
>     >> from that representing the whole rx.  With chunked, what was
>     received
>     >may
>     >> contain multiple (and partial...) chunking transfer encoding inline
>     >> headers.  Lws strips these before giving it to you by making a
>     >callback to
>     >> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ for each whole or partial
>     >payload
>     >> region in the buffer.
>     >>
>     >>  - since the writable callback is only good for one write each
>     time,
>     >but
>     >> in chunked case lws_http_client_read() may spatter you with many
>     >pieces
>     >> reflecting how the chunking was sent, you should coalesce them
>     before
>     >> writing
>     >>
>     >> -Andy
>     >>
>     >> >
>     >> >[2017/03/06 08:00:00:7171] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_ESTABLISHED_CLIENT_HTTP: 200
>     >> >[2017/03/06 08:00:00:7181] INFO: lws_header_table_detach: wsi
>     >0301F760:
>     >> >ah
>     >> >00EE43B8 (tsi=0, count = 2)
>     >> >[2017/03/06 08:00:00:7191] INFO: lws_header_table_detach: wsi
>     >0301F760:
>     >> >ah
>     >> >00EE43B8 (tsi=0, count = 1)
>     >> >[2017/03/06 08:00:00:7201] NOTICE:
>     >> >lws_client_interpret_server_handshake:
>     >> >client connection up
>     >> >[2017/03/06 08:00:00:7201] INFO: ssl buffered read
>     >> >[2017/03/06 08:00:00:7211] INFO: chunk 32768
>     >> >[2017/03/06 08:00:00:7221] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 701
>     >> >[2017/03/06 08:00:00:7241] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7261] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7261] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7271] INFO: http: (0301F450)
>     >> >LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>     >> >[2017/03/06 08:00:00:7271] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7281] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7281] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7291] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7291] INFO: http: (0301F450)
>     >> >LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>     >> >[2017/03/06 08:00:00:7301] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7301] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7311] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7321] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7331] INFO: http: (0301F450)
>     >> >LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>     >> >[2017/03/06 08:00:00:7341] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7371] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7371] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7381] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7391] INFO: http: (0301F450)
>     >> >LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>     >> >[2017/03/06 08:00:00:7391] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7401] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7401] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7401] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7411] INFO: http: (0301F450)
>     >> >LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>     >> >[2017/03/06 08:00:00:7411] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7421] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7431] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7431] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7431] INFO: http: (0301F450)
>     >> >LWS_CALLBACK_HTTP_WRITEABLE: v=02FFB770, ctx=00ED1CF0
>     >> >[2017/03/06 08:00:00:7441] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7441] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7451] INFO: http: (0301F760)
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 1409
>     >> >[2017/03/06 08:00:00:7451] INFO: lws_read: read_ok, used 0
>     >> >[2017/03/06 08:00:00:7461] NOTICE: lws_server_socket_service: wsi
>     >> >0301F450
>     >> >read 0
>     >> >[2017/03/06 08:00:00:7501] INFO: lws_server_socket_service: read 0
>     >len
>     >> >[2017/03/06 08:00:00:7511] ERR: 169.254.225.161 - -
>     >> >[06/Mar/2017:08:00:00
>     >> >-0800] "POST /rest/send http/1.1" 200 8976 Mozilla/5.0 (Windows NT
>     >> >10.0;
>     >> >WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87
>     >> >Safari/537.36
>     >> >[2017/03/06 08:00:00:7521] INFO: lws_close_free_wsi: real
>     >> >just_kill_connection: 0301F760 (sockfd 560)
>     >> >[2017/03/06 08:00:00:7531] INFO: remove_wsi_socket_from_fds:
>     >removing
>     >> >same
>     >> >prot wsi 0301F760
>     >> >[2017/03/06 08:00:00:7531] INFO: lws_close_free_wsi: shutting down
>     >> >connection: 0301F450 (sock 500, state 3)
>     >> >[2017/03/06 08:00:00:7541] INFO: lws_close_free_wsi: real
>     >> >just_kill_connection: 0301F450 (sockfd 500)
>     >> >[2017/03/06 08:00:00:7541] INFO: remove_wsi_socket_from_fds:
>     >removing
>     >> >same
>     >> >prot wsi 0301F450
>     >> >[2017/03/06 08:00:00:7551] INFO: ah det due to close
>     >> >[2017/03/06 08:00:00:7551] INFO: lws_header_table_detach: wsi
>     >0301F450:
>     >> >ah
>     >> >00EE35A0 (tsi=0, count = 1)
>     >> >[2017/03/06 08:00:00:7561] INFO: lws_header_table_detach: wsi
>     >0301F450:
>     >> >ah
>     >> >00EE35A0 (tsi=0, count = 0)
>     >> >[2017/03/06 08:00:00:7571] INFO: ah det due to close
>     >> >[2017/03/06 08:00:00:7571] INFO: lws_header_table_detach: wsi
>     >0301F760:
>     >> >ah
>     >> >00000000 (tsi=0, count = 0)
>     >> >[2017/03/06 08:00:00:7581] INFO: lws_header_table_detach: wsi
>     >0301F760:
>     >> >ah
>     >> >00000000 (tsi=0, count = 0)
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >
>     >> >On Sun, Mar 5, 2017 at 8:43 PM Andy Green <andy at warmcat.com
>     <mailto:andy at warmcat.com>> wrote:
>     >> >
>     >> >>
>     >> >>
>     >> >> On 03/05/2017 11:55 PM, Joel Winarske wrote:
>     >> >> > Hi Andy,
>     >> >> >
>     >> >> > I'm seeing a problem with the client.  lws_http_client_read()
>     >> >returns
>     >> >> > a value in both RECEIVE_CLIENT_HTTP callbacks.  I would expect
>     >it
>     >> >> > would only return a value for chunked given the size.  See
>     >anything
>     >> >> > suspicious here?
>     >> >> >
>     >> >> > [2017/03/05 07:51:40:9226] NOTICE: lws_ssl_client_connect2:
>     >> >> > SSL_connect says 1
>     >> >> > [2017/03/05 07:51:41:4546] INFO: lws_ensure_user_space:
>     031651B0
>     >> >> > protocol 02E25628
>     >> >> > [2017/03/05 07:51:41:4556] INFO: lws_ensure_user_space:
>     031651B0
>     >> >> > protocol pss 11308, user_space=03173C10
>     >> >> > [2017/03/05 07:51:41:4566] NOTICE:
>     >> >> > lws_client_interpret_server_handshake: incoming content length
>     >9506
>     >> >> >     connection: keep-alive
>     >> >> >     http/1.1  200 OK
>     >> >> >     pragma: no-cache
>     >> >> >     cache-control: no-cache
>     >> >> >     content-length: 9506
>     >> >> >     content-type: application/json; charset=utf-8
>     >> >> >     date: Sun, 05 Mar 2017 15:51:38 GMT
>     >> >> >     expires: -1
>     >> >> >     server: Microsoft-IIS/8.5
>     >> >> > [2017/03/05 07:51:41:4626] INFO: lws_header_table_detach: wsi
>     >> >> > 031651B0: ah 00E53258 (tsi=0, count = 2)
>     >> >> > [2017/03/05 07:51:41:4636] INFO: lws_header_table_detach: wsi
>     >> >> > 031651B0: ah 00E53258 (tsi=0, count = 1)
>     >> >> > [2017/03/05 07:51:41:4636] NOTICE:
>     >> >> > lws_client_interpret_server_handshake: client connection up
>     >> >> > [2017/03/05 07:51:41:4636] INFO: ssl buffered read
>     >> >> > [2017/03/05 07:51:41:4646] INFO: http: (031651B0)
>     >> >> > LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ (chunked): 4768
>     >> >> > [2017/03/05 07:51:41:4646] INFO: http: (031651B0)
>     >> >> > LWS_CALLBACK_RECEIVE_CLIENT_HTTP (unchunked) 4768
>     >> >>
>     >> >> How about only taking data from
>     >> >LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ,
>     >> >> for both chunked and unchunked?
>     >> >>
>     >> >> -Andy
>     >> >>
>     >> >> > [2017/03/05 07:51:41:4656] INFO: lws_close_free_wsi: shutting
>     >down
>     >> >SSL
>     >> >> > connection: 031651B0 (ssl 0317AD78, sock 560, state 6)
>     >> >> > [2017/03/05 07:51:41:4666] INFO: lws_close_free_wsi: real
>     >> >> > just_kill_connection: 031651B0 (sockfd 560)
>     >> >> > [2017/03/05 07:51:41:4676] INFO: remove_wsi_socket_from_fds:
>     >> >removing
>     >> >> > same prot wsi 031651B0
>     >> >> > [2017/03/05 07:51:41:4676] INFO: ah det due to close
>     >> >> > [2017/03/05 07:51:41:4686] INFO: lws_header_table_detach: wsi
>     >> >> > 031651B0: ah 00000000 (tsi=0, count = 1)
>     >> >> > [2017/03/05 07:51:41:4686] INFO: lws_header_table_detach: wsi
>     >> >> > 031651B0: ah 00000000 (tsi=0, count = 1)
>     >> >> >
>     >> >> >
>     >> >> >
>     >> >> > _______________________________________________
>     >> >> > Libwebsockets mailing list
>     >> >> > Libwebsockets at ml.libwebsockets.org
>     <mailto:Libwebsockets at ml.libwebsockets.org>
>     >> >> > https://libwebsockets.org/mailman/listinfo/libwebsockets
>     <https://libwebsockets.org/mailman/listinfo/libwebsockets>
>     >> >>
>     >> >>
>     >>
>
>




More information about the Libwebsockets mailing list