[Libwebsockets] What is the proper way to keep stream connection?

Andy Green andy at warmcat.com
Sun Apr 28 06:07:58 CEST 2019

On April 28, 2019 1:31:31 AM GMT+01:00, Kun Zhao <kunzhao77 at gmail.com> wrote:
>Hi Andy,
>I have a HTTP client which receives data from a server. The connection
>kept alive and the server sends data to the client periodically. The
>receives data for a while then closed the connection. After some
>I found that the problem is here:
>the ah->assigned never updates and the session eventually timed out
>and the client closed the connection.

ah->assigned should never update... this code represents an lws policy that user code must not hold onto an ah for a long period, the limit's like 5 minutes.

ah are 'detachable' heap-allocated structs that contain parsed http headers... the're a few KB each but can be set larger at context creation time; an ah pool can also be defined there based on how scarce memory is in your system.  On h1, connections acquire an ah or go on a waiting list for one before the request (server) or response (client) headers can be parsed.

On h2, all the care about waiting lists and pools was turned on its head... since h2 muxes control packets and an arbitrary number of http streams on one connection, dealing with h2 means you cannot just defer reading from the connection... you will deadlock being unable to send anything because you deferred reading it to find the necessary tx credit update control packets.  So if the h2 connection says it wants to open 10 x http streams concurrently, which is pretty typical, lws must find or allocate 10 x ah suddenly... we can't tolerate using the ah waiting list.

For this reason ah are required to be freed ASAP and user code that keeps them is regarded as a bug by lws.  For connections that upgrade to ws, this is done by lws for you, the ah is detached just after ESTABLISHED is called.  This is why if you care about the http headers on a ws connection, you must query them at ESTABLISHED or keep copies of ones you care about at that point.

For long-lived http connections, you can detach the ah manually, again after querying or copying critical info from it


For h1 that and the wsi timeout is all lws needs you to do... intermediaries or the remote server that understand h1 might enforce their own idle timeouts though.  For h2 it's more complicated again... there are two idle timeouts in play, one for a logical stream wsi, and one for the 'nwsi' network wsi, the h2 socket connection that carries one or more streams.  When you write something, both get 30s of life.  So if any stream writes anything, the underlying nwsi keeps getting life.  If an individual stream is idle, it will quickly get closed.

With SSE and ws-over-h2, it introduced a concept of an 'immortal stream'... an nwsi carrying at least one immortal stream is itself no longer subject to timeouts.  This is to deliver the necessary ws semantics over h2, where ws streams can idle forever by default.  This works ok vs the ah assignation timeout because nwsi don't have an ah themselves, and the upgraded-to-ws stream dropped their ah.

Since you're already doing the keepalive stuff, it sounds like you just need to drop the ah before entering your long wait.


>What is the proper way to update the ah->assigned time stamp to keep
>connection open?
>Following is the response header from the server.
>HTTP/1.1 200 OK
>Server: openresty/
>Date: Sat, 27 Apr 2019 21:11:14 GMT
>Content-Type: application/octet-stream
>Transfer-Encoding: chunked
>Connection: close
>Access-Control-Allow-Headers: Authorization, Content-Type,
>Accept-Datetime-Format, OANDA-Agent, ETag
>Access-Control-Allow-Methods: PUT, PATCH, POST, GET, OPTIONS, DELETE
>Access-Control-Allow-Origin: *
>Access-Control-Expose-Headers: ETag, RequestID
>RequestID: 799161546957200243
>Following is the logs:
>[2019/04/27 19:05:01:8002] NOTICE: ah excessive hold: wsi
>  peer address:
>  ah pos 258
>[2019/04/27 19:05:01:8032] NOTICE:    connection: = close
>[2019/04/27 19:05:01:8042] NOTICE:    http/1.1  = 200 OK
>[2019/04/27 19:05:01:8052] NOTICE:    content-type: =
>[2019/04/27 19:05:01:8072] NOTICE:    date: = Sat, 27 Apr 2019 23:56:03
>[2019/04/27 19:05:01:8092] NOTICE:    access-control-allow-origin: = *
>[2019/04/27 19:05:01:8102] NOTICE:    server: = openresty/
>[2019/04/27 19:05:01:8122] NOTICE:    transfer-encoding: = chunked
>[2019/04/27 19:05:01:8132] INFO: __lws_header_table_detach: wsi
>00000000122CFC40: ah 000000001230F9C0 (tsi=0, count = 1)
>[2019/04/27 19:05:01:8152] DEBUG: __lws_header_table_detach: wsi
>00000000122CFC40: ah held 544s, role/state 0x10000000 0x117,
>[2019/04/27 19:05:01:8182] INFO: __lws_header_table_detach: nobody
>[2019/04/27 19:05:01:8202] INFO: _lws_destroy_ah: freed ah
>: pool length 0
>[2019/04/27 19:05:01:8222] INFO: __lws_header_table_detach: wsi
>00000000122CFC40: ah 000000001230F9C0 (tsi=0, count = 0)
>[2019/04/27 19:05:01:8242] INFO: __lws_close_free_wsi:
>caller: excessive ah
>[2019/04/27 19:05:01:8262] DEBUG: __lws_close_free_wsi: real
>just_kill_connection: 00000000122CFC40 (sockfd 260)
>[2019/04/27 19:05:01:8282] DEBUG: __remove_wsi_socket_from_fds:
>wsi=00000000122CFC40, skt=260, fds pos=0, end guy pos=1, endfd=0
>[2019/04/27 19:05:01:8312] DEBUG: lwsi_set_state(00000000122CFC40,
>Kun Zhao

More information about the Libwebsockets mailing list