[Libwebsockets] HTTP client receive behavior changed?

Andy Green andy at warmcat.com
Fri Apr 5 23:08:26 CEST 2019



On April 6, 2019 4:29:14 AM GMT+08:00, Kun Zhao <kunzhao77 at gmail.com> wrote:
>The COMPLETED_CLIENT_HTTP is called before the last
>RECEIVE_CLIENT_HTTP.
>Here is the callback sequence.
>
>[2019/04/05 15:18:16:7170] INFO: lws_client_interpret_server_handshake:
>client connection up
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>******** LWS_CALLBACK_COMPLETED_CLIENT_HTTP
>******** LWS_CALLBACK_RECEIVE_CLIENT_HTTP
>[2019/04/05 15:18:16:7187] INFO: lws_http_transaction_completed_client:
>wsi: 0x7fd6c8000b60, wsi_eff: 0x7fd6c8000b60 (http)
>[2019/04/05 15:18:16:7187] INFO: lws_http_transaction_completed_client:
>nothing pipelined waiting
>******** LWS_CALLBACK_CLOSED_CLIENT_HTTP
>[2019/04/05 15:18:22:0037] INFO: wsi 0x7fd6c8000b60: TIMEDOUT WAITING
>on 27
>(did hdr 0, ah 0x7fd6c8000e60, wl 0, pfd events 1) 1554495502 vs 5
>[2019/04/05 15:18:22:0037] INFO: __lws_close_free_wsi: 0x7fd6c8000b60:
>caller: timeout
>
>
>On Fri, Apr 5, 2019 at 2:28 PM Kun Zhao <kunzhao77 at gmail.com> wrote:
>
>> Hi Andy,
>>
>> I updated libwebsockets from 3.0 to master branch today. I noticed a
>> behavior change in HTTP client. HTTP client used to receive data in
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ callback. Now the
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ is never called (I guess for
>> un-chunked data) and the data is provided in
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP.
>>
>> Just want to check with you if this is the correct behavior because
>of all
>> the minimal examples and test client are still read data in
>> LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ  callback.

Several minimal clients are built and run in CI, so if something like that was true the CI build breaks.

What actually happens is when http rx comes, the user code is notified with LWS_CALLBACK_RECEIVE_CLIENT_HTTP.  You can take the raw h1 data there, but with h1, at the decision of the server (many cgis will enforce it for their own convenience in being able to emit without knowing the content-length beforehand) the payload rx may be chunked.  Chunk headers are then inserted at arbitrary intervals inline with the payload.

Additionally, in many cases, there will be onward transfer to another socket with some form of the incoming data... eg, proxying bulk data from the h1 server to an onward ws link.  When you want to consume the pending data is then something the onward link writeability decides.  (Lws supports h1 / h2 <-> h1 reverse proxying natively using this stuff... in fact when you visit libwebsockets.org git, it's transparently delivered to h1 or h2 clients via an lws reverse proxy link from a gitohashi process that's actually serving h1 on a unix domain socket on the same machine.)

For those reasons the rx processing is split between the notification raw rx http payload stuff is available, at which point you can choose to leave it buffered by lws and turn off rx flow control and return to the event loop until you can consume it, or you can immediately dechunk it and consume it using the lws_http_client_read() api...

https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/http-client/minimal-http-client/minimal-http-client.c#n57-66

... which calls back LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ for each dechunked block of data before returning.

-Andy

>> Thank,
>> Kun
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20190406/f8f1630d/attachment.htm>


More information about the Libwebsockets mailing list