{"schema":"libjg2-1", "vpath":"/git/", "avatar":"/git/avatar/", "alang":"", "gen_ut":1748971573, "reponame":"libwebsockets", "desc":"libwebsockets lightweight C networking library", "owner": { "name": "Andy Green", "email": "andy@warmcat.com", "md5": "c50933ca2aa61e0fe2c43d46bb6b59cb" },"url":"https://libwebsockets.org/repo/libwebsockets", "f":3, "items": [ { "schema":"libjg2-1", "oid":{ "oid": "b3576e7205d630c9e25df62b971db890f3b7c40c", "alias": []},"tree": [ { "name": "plugins","mode": "16384", "size":0}, { "name": "protocols","mode": "16384", "size":0}, { "name": "system","mode": "16384", "size":0}, { "name": "README.md","mode": "33188", "size":18587}, { "name": "policy-common.c","mode": "33188", "size":7659}, { "name": "policy-json.c","mode": "33188", "size":20032}, { "name": "private-lib-secure-streams.h","mode": "33188", "size":10345}, { "name": "secure-streams-client.c","mode": "33188", "size":12732}, { "name": "secure-streams-process.c","mode": "33188", "size":13096}, { "name": "secure-streams-serialize.c","mode": "33188", "size":21147}, { "name": "secure-streams.c","mode": "33188", "size":14303}],"s":{"c":1748971573,"u": 3692}} ,{"schema":"libjg2-1", "cid":"ddc48efd1876932fe61bafa865e0a623", "oid":{ "oid": "b3576e7205d630c9e25df62b971db890f3b7c40c", "alias": []},"blobname": "lib/secure-streams/README.md", "blob": "# Secure Streams\n\nSecure Streams is a client api that strictly separates payload from any metadata.\nThat includes the endpoint address for the connection, the tls CA and even the\nprotocol used to connect to the endpoint.\n\nThe user api just receives and transmits payload, and receives advisory connection\nstate information.\n\nThe details about how the connections for different types of secure stream should\nbe made are held in JSON \u0022policy database\u0022 initially passed in to the context\ncreation, but able to be updated from a remote copy.\n\n\n\n## Convention for rx and tx callback return\n\nFunction|Return|Meaning\n---|---|---\ntx|0|Send the amount of `buf` stored in `*len`\ntx|\u003e0|Do not send anything\ntx|\u003c0|Finished with stream\nrx|\u003e\u003d0|accepted\nrx|\u003c0|Finished with stream\n\n\n# JSON Policy Database\n\nExample JSON policy... formatting is shown for clarity but whitespace can be\nomitted in the actual policy.\n\nOrdering is not critical in itself, but forward references are not allowed,\nthings must be defined before they are allowed to be referenced later in the\nJSON.\n\n\n```\n{\n\t\u0022release\u0022: \u002201234567\u0022,\n\t\u0022product\u0022: \u0022myproduct\u0022,\n\t\u0022schema-version\u0022: 1,\n\t\u0022retry\u0022: [{\n\t\t\u0022default\u0022: {\n\t\t\t\u0022backoff\u0022: [1000, 2000, 3000, 5000, 10000],\n\t\t\t\u0022conceal\u0022: 5,\n\t\t\t\u0022jitterpc\u0022: 20\n\t\t}\n\t}],\n\t\u0022certs\u0022: [{\n\t\t\u0022isrg_root_x1\u0022: \u0022MIIFazCCA1OgAw...AnX5iItreGCc\u003d\u0022\n\t}, {\n\t\t\u0022LEX3_isrg_root_x1\u0022: \u0022MIIFjTCCA3WgAwIB...WEsikxqEt\u0022\n\t}],\n\t\u0022trust_stores\u0022: [{\n\t\t\u0022le_via_isrg\u0022: [\u0022isrg_root_x1\u0022, \u0022LEX3_isrg_root_x1\u0022]\n\t}],\n\t\u0022s\u0022: [{\n\t\t\u0022mintest\u0022: {\n\t\t\t\u0022endpoint\u0022: \u0022warmcat.com\u0022,\n\t\t\t\u0022port\u0022: 4443,\n\t\t\t\u0022protocol\u0022: \u0022h1get\u0022,\n\t\t\t\u0022aux\u0022: \u0022index.html\u0022,\n\t\t\t\u0022plugins\u0022: [],\n\t\t\t\u0022tls\u0022: true,\n\t\t\t\u0022opportunistic\u0022: true,\n\t\t\t\u0022retry\u0022: \u0022default\u0022,\n\t\t\t\u0022tls_trust_store\u0022: \u0022le_via_isrg\u0022\n\t\t}\n\t}]\n}\n```\n\n### `Release`\n\nIdentifies the policy version\n\n### `Product`\n\nIdentifies the product the policy should apply to\n\n### `Schema-version`\n\nThe minimum version of the policy parser required to parse this policy\n\n### `via-socks5`\n\nOptional redirect for Secure Streams client traffic through a socks5\nproxy given in the format `address:port`, eg, `127.0.0.1:12345`.\n\n### `retry`\n\nA list of backoff schemes referred to in the policy\n\n### `backoff`\n\nAn array of ms delays for each retry in turn\n\n### `conceal`\n\nThe number of retries to conceal from higher layers before giving errors. If\nthis is larger than the number of times in the backoff array, then the last time\nis used for the extra delays\n\n### `jitterpc`\n\nPercentage of the delay times mentioned in the backoff array that may be\nrandomly added to the figure from the array. For example with an array entry of\n1000ms, and jitterpc of 20%, actual delays will be chosen randomly from 1000ms\nthrough 1200ms. This is to stop retry storms triggered by a single event like\nan outage becoming synchronized into a DoS.\n\n### `certs`\n\nCertificates needed for validation should be listed here each with a name. The\nformat is base64 DER, which is the same as the part of PEM that is inside the\nstart and end lines.\n\n### `trust_stores`\n\nChains of certificates given in the `certs` section may be named and described\ninside the `trust_stores` section. Each entry in `trust_stores` is created as\na vhost + tls context with the given name. Stream types can later be associated\nwith one of these to enforce validity checking of the remote server.\n\nEntries should be named using \u0022name\u0022 and the stack array defined using \u0022stack\u0022\n\n### `s`\n\nThese are an array of policies for the supported stream type names.\n\n### `endpoint`\n\nThe DNS address the secure stream should connect to.\n\nThis may contain string symbols which will be replaced with the\ncorresponding streamtype metadata value at runtime. Eg, if the\nstreamtype lists a metadata name \u0022region\u0022, it's then possible to\ndefine the endpoint as, eg, `${region}.mysite.com`, and before\nattempting the connection setting the stream's metadata item\n\u0022region\u0022 to the desired value, eg, \u0022uk\u0022.\n\nIf the endpoint string begins with `+`, then it's understood to\nmean a connection to a Unix Domain Socket, for Linux `+@` means\nthe following Unix Domain Socket is in the Linux Abstract\nNamespace and doesn't have a filesystem footprint. This is only\nsupported on unix-type and windows platforms and when lws was\nconfigured with `-DLWS_UNIX_SOCK\u003d1`\n\n### `port`\n\nThe port number as an integer on the endpoint to connect to\n\n### `protocol`\n\nThe wire protocol to connect to the endpoint with. Currently supported\nstreamtypes are\n\n|Wire protocol|Description|\n|---|---|\n|h1|http/1|\n|h2|http/2|\n|ws|http/1 Websockets|\n|mqtt|mqtt 3.1.1|\n|raw||\n\nRaw protocol is a bit different than the others in that there is no protocol framing,\nwhatever is received on the connection is passed to the user rx callback and whatever\nthe tx callback provides is issued on to the connection. Because tcp can be\narbitrarily fragmented by any intermediary, such streams have to be regarded as an\nordered bytestream that may be fragmented at any byte without any meaning in terms\nof message boundaries, for that reason SOM and EOM are ignored with raw.\n\n### `plugins`\n\nArray of plugin names to apply to the stream, if any\n\n### `tls`\n\nSet to `true` to enforce the stream travelling in a tls tunnel\n\n### `client cert`\n\nSet if the stream needs to authenticate itself using a tls client certificate.\nSet to the certificate index counting from 0+. The certificates are managed\nusing lws_sytstem blobs.\n\n### `opportunistic`\n\nSet to `true` if the connection may be left dropped except when in use\n\n### `nailed_up`\n\nSet to `true` to have lws retry if the connection carrying this stream should\never drop.\n\n### `retry`\n\nThe name of the policy described in the `retry` section to apply to this\nconnection for retry + backoff\n\n### `tls_trust_store`\n\nThe name of the trust store described in the `trust_stores` section to apply\nto validate the remote server cert.\n\n## http transport\n\n### `http_method`\n\nHTTP method to use with http-related protocols, like GET or POST.\nNot required for ws.\n\n### `http_expect`\n\nOptionally indicates that success for HTTP transactions using this\nstreamtype is different than the default 200 - 299.\n\nEg, you may choose to set this to 204 for Captive Portal Detect usage\nif that's what you expect the server to reply with to indicate\nsuccess. In that case, anything other than 204 will be treated as a\nconnection failure.\n\n### `http_fail_redirect`\n\nSet to `true` if you want to fail the connection on meeting an\nhttp redirect. This is needed to, eg, detect Captive Portals\ncorrectly. Normally, if on https, you would want the default behaviour\nof following the redirect.\n\n### `http_url`\n\nUrl path to use with http-related protocols\n\nThe URL path can include metatadata like this\n\n\u0022/mypath?whatever\u003d${metadataname}\u0022\n\n${metadataname} will be replaced by the current value of the\nsame metadata name. The metadata names must be listed in the\n\u0022metadata\u0022: [ ] section.\n\n### `http_auth_header`\n\nThe name of the header that takes the auth token, with a trailing ':', eg\n\n```\n \u0022http_auth_header\u0022: \u0022authorization:\u0022\n```\n\n### `http_dsn_header`\n\nThe name of the header that takes the dsn token, with a trailing ':', eg\n\n```\n \u0022http_dsn_header\u0022: \u0022x-dsn:\u0022\n```\n\n### `http_fwv_header`\n\nThe name of the header that takes the firmware version token, with a trailing ':', eg\n\n```\n \u0022http_fwv_header\u0022: \u0022x-fw-version:\u0022\n```\n\n### `http_devtype_header`\n\nThe name of the header that takes the device type token, with a trailing ':', eg\n\n```\n \u0022http_devtype_header\u0022: \u0022x-device-type:\u0022\n```\n\n### `http_auth_preamble`\n\nAn optional string that precedes the auth token, eg\n\n```\n \u0022http_auth_preamble\u0022: \u0022bearer \u0022\n```\n\n### `auth_hexify`\n\nConvert the auth token to hex ('A' -\u003e \u002241\u0022) before transporting. Not necessary if the\nauth token is already in printable string format suitable for transport. Needed if the\nauth token is a chunk of 8-bit binary.\n\n### `nghttp2_quirk_end_stream`\n\nSet this to `true` if the peer server has the quirk it won't send a response until we have\nsent an `END_STREAM`, even though we have sent headers with `END_HEADERS`.\n\n### `h2q_oflow_txcr`\n\nSet this to `true` if the peer server has the quirk it sends an maximum initial tx credit\nof 0x7fffffff and then later increments it illegally.\n\n### `http_multipart_name`\n\nIndicates this stream goes out using multipart mime, and provides the name part of the\nmultipart header\n\n### `http_multipart_filename`\n\nIndicates this stream goes out using multipart mime, and provides the filename part of the\nmultipart header\n\n### `http_multipart_content_type`\n\nThe `content-type` to mark up the multipart mime section with if present\n\n### `http_www_form_urlencoded`\n\nIndicate the data is sent in `x-www-form-urlencoded` form\n\n### `rideshare`\n\nFor special cases where one logically separate stream travels with another when using this\nprotocol. Eg, a single multipart mime transaction carries content from two or more streams.\n\n## ws transport\n\n### `ws_subprotocol`\n\nName of the ws subprotocol to use.\n\n### `ws_binary`\n\nUse if the ws messages are binary\n\n## MQTT transport\n\n### `mqtt_topic`\n\nSet the topic this streamtype uses for writes\n\n### `mqtt_subscribe`\n\nSet the topic this streamtype subscribes to\n\n### `mqtt qos`\n\nSet the QOS level for this streamtype\n\n### `mqtt_keep_alive`\n\n16-bit number representing MQTT keep alive for the stream.\n\nThis is applied at connection time... where different streams may bind to the\nsame underlying MQTT connection, all the streams should have an identical\nsetting for this.\n\n### `mqtt_clean_start`\n\nSet to true if the connection should use MQTT's \u0022clean start\u0022 feature.\n\nThis is applied at connection time... where different streams may bind to the\nsame underlying MQTT connection, all the streams should have an identical\nsetting for this.\n\n### `mqtt_will_topic`\n\nSet the topic of the connection's will message, if any (there is none by default).\n\nThis is applied at connection time... where different streams may bind to the\nsame underlying MQTT connection, all the streams should have an identical\nsetting for this.\n\n### `mqtt_will_message`\n\nSet the content of the connect's will message, if any (there is none by default).\n\nThis is applied at connection time... where different streams may bind to the\nsame underlying MQTT connection, all the streams should have an identical\nsetting for this.\n\n### `mqtt_will_qos`\n\nSet the QoS of the will message, if any (there is none by default).\n\nThis is applied at connection time... where different streams may bind to the\nsame underlying MQTT connection, all the streams should have an identical\nsetting for this.\n\n### `mqtt_will_retain`\n\nSet to true if the connection should use MQTT's \u0022will retain\u0022 feature, if there\nis a will message (there is none by default).\n\nThis is applied at connection time... where different streams may bind to the\nsame underlying MQTT connection, all the streams should have an identical\nsetting for this.\n\n## Loading and using updated remote policy\n\nIf the default, hardcoded policy includes a streamtype `fetch_policy`,\nduring startup when lws_system reaches the POLICY state, lws will use\na Secure Stream of type `fetch_policy` to download, parse and update\nthe policy to use it.\n\nThe secure-streams-proxy minimal example shows how this is done and\nfetches its real policy from warmcat.com at startup using the built-in\none.\n\n## Applying streamtype policy overlays\n\nThis is intended for modifying policies at runtime for testing, eg, to\nforce error paths to be taken. After the main policy is processed, you\nmay parse additional, usually smaller policy fragments on top of it.\n\nWhere streamtype names in the new fragment already exist in the current\nparsed policy, the settings in the fragment are applied over the parsed\npolicy, overriding settings. There's a simple api to enable this by\ngiving it the override JSON in one string\n\n```\nint\nlws_ss_policy_overlay(struct lws_context *context, const char *overlay);\n```\n\nbut there are also other apis available that can statefully process\nlarger overlay fragments if needed.\n\nAn example overlay fragment looks like this\n\n```\n\t{ \u0022s\u0022: [{ \u0022captive_portal_detect\u0022: {\n\t\t\u0022endpoint\u0022: \u0022google.com\u0022,\n\t\t\u0022http_url\u0022: \u0022/\u0022,\n\t\t\u0022port\u0022: 80\n\t}}]}\n```\n\nie the overlay fragment completely follows the structure of the main policy,\njust misses out anything it doesn't override.\n\nCurrently ONLY streamtypes may be overridden.\n\nYou can see an example of this in use in `minimal-secure-streams` example\nwhere `--force-portal` and `--force-no-internet` options cause the captive\nportal detect streamtype to be overridden to force the requested kind of\noutcome.\n\n## Captive Portal Detection\n\nIf the policy contains a streamtype `captive_portal_detect` then the\ntype of transaction described there is automatically performed after\nacquiring a DHCP address to try to determine the captive portal\nsituation.\n\n```\n\t\t\u0022captive_portal_detect\u0022: {\n \u0022endpoint\u0022: \u0022connectivitycheck.android.com\u0022,\n \u0022port\u0022: 80,\n \u0022protocol\u0022: \u0022h1\u0022,\n \u0022http_method\u0022: \u0022GET\u0022,\n \u0022http_url\u0022: \u0022generate_204\u0022,\n \u0022opportunistic\u0022: true,\n \u0022http_expect\u0022: 204,\n\t\t\t\u0022http_fail_redirect\u0022: true\n }\n```\n\n## Stream serialization and proxying\n\nBy default Secure Streams expects to make the outgoing connection described in\nthe policy in the same process / thread, this suits the case where all the\nparticipating clients are in the same statically-linked image.\n\nIn this case the `lws_ss_` apis are fulfilled locally by secure-streams.c and\npolicy.c for policy lookups.\n\nHowever it also supports serialization, where the SS api can be streamed over\nanother transport such as a Unix Domain Socket connection. This suits the case\nwhere the clients are actually in different processes in, eg, Linux or Android.\n\nIn those cases, you run a proxy process (minimal-secure-streams-proxy) that\nlistens on a Unix Domain Socket and is connected to by one or more other\nprocesses that pass their SS API activity to the proxy for fulfilment (or\nonward proxying).\n\nIn this case the proxy uses secure-streams.c and policy.c as before to fulfil\nthe inbound proxy streams, but uses secure-streams-serialize.c to serialize and\ndeserialize the proxied SS API activity. The proxy clients define\nLWS_SS_USE_SSPC either very early in their sources before the includes, or on\nthe compiler commandline... this causes the lws_ss_ apis to be replaced at\npreprocessor time with lws_sspc_ equivalents. These serialize the api action\nand pass it to the proxy over a Unix Domain Socket for fulfilment, the results\nand state changes etc are streamed over the Unix Domain Socket and presented to\nthe application exactly the same as if it was being fulfilled locally.\n\nTo demonstrate this, some minimal examples, eg, minimal-secure-streams and\nmimimal-secure-streams-avs build themselves both ways, once with direct SS API\nfulfilment and once with Unix Domain Socket proxying and -client appended on the\nexecutable name. To test the -client variants, run minimal-secure-streams-proxy\non the same machine.\n\n## Complicated scenarios with secure streams proxy\n\nAs mentioned above, Secure Streams has two modes, by default the application\ndirectly parses the policy and makes the outgoing connections itself.\nHowever when configured at cmake with\n\n```\n-DLWS_WITH_SOCKS5\u003d1 -DLWS_WITH_SECURE_STREAMS\u003d1 -DLWS_WITH_SECURE_STREAMS_PROXY_API\u003d1 -DLWS_WITH_MINIMAL_EXAMPLES\u003d1\n```\n\nand define `LWS_SS_USE_SSPC` when building the application, applications forward\ntheir network requests to a local or remote SS proxy for fulfilment... and only\nthe SS proxy has the system policy. By default, the SS proxy is on the local\nmachine and is connected to via a Unix Domain Socket, but tcp links are also\npossible. (Note the proxied traffic is not encrypyed by default.)\n\nUsing the configuration above, the example SS applications are built two ways,\nonce for direct connection fulfilment (eg, `./bin/lws-minimal-secure-streams`),\nand once with `LWS_SS_USE_SSPC` also defined so it connects via an SS proxy,\n(eg, `./bin/lws-minimal-secure-streams-client`).\n\n## Testing an example scenario with SS Proxy and socks5 proxy\n\n```\n [ SS application ] --- tcp --- [ socks 5 proxy ] --- tcp --- [ SS proxy ] --- internet\n```\n\nIn this scenario, everything is on localhost, the socks5 proxy listens on :1337 and\nthe SS proxy listens on :1234. The SS application connects to the socks5\nproxy to get to the SS proxy, which then goes out to the internet\n\n### 1 Start the SS proxy\n\nTell it to listen on lo interface on port 1234 \n\n```\n$ ./bin/lws-minimal-secure-streams-proxy -p 1234 -i lo\n```\n\n### 2 Start the SOCKS5 proxy\n\n```\n$ ssh -D 1337 -N -v localhost\n```\n\nThe -v makes connections to the proxy visible in the terminal for testing\n\n### 3 Run the SS application\n\nThe application is told to make all connections via the socks5 proxy at\n127.0.0.1:1337, and to fulfil its SS connections via an SS proxy, binding\nconnections to 127.0.0.1 (ipv4 lo interface, -1), to 127.0.0.1:1234 (-a/-p).\n\n```\nsocks_proxy\u003d127.0.0.1:1337 ./bin/lws-minimal-secure-streams-client -p 1234 -i 127.0.0.1 -a 127.0.0.1\n```\n\nYou can confirm this goes through the ssh socks5 proxy to get to the SS proxy\nand fulfil the connection.\n\n## Using static policies\n\nIf one of your targets is too constrained to make use of dynamic JSON policies, but\nusing SS and the policies is attractive for wider reasons, you can use a static policy\nbuilt into the firmware for the constrained target.\n\nThe secure-streams example \u0022policy2c\u0022 (which runs on the build machine, not the device)\n\nhttps://libwebsockets.org/git/libwebsockets/tree/minimal-examples/secure-streams/minimal-secure-streams-policy2c\n\naccepts a normal JSON policy on stdin, and emits a C code representation that can be\nincluded directly in the firmware.\n\nhttps://libwebsockets.org/git/libwebsockets/tree/minimal-examples/secure-streams/minimal-secure-streams-staticpolicy/static-policy.h\n\nUsing this technique it's possible to standardize on maintaining JSON policies across a\nrange of devices with different contraints, and use the C conversion of the policy on devices\nthat are too small.\n\nThe Cmake option `LWS_WITH_SECURE_STREAMS_STATIC_POLICY_ONLY` should be enabled to use this\nmode, it will not build the JSON parser (and the option for LEJP can also be disabled if\nyou're not otherwise using it, saving an additional couple of KB).\n\nNotice policy2c example tool must be built with `LWS_ROLE_H1`, `LWS_ROLE_H2`, `LWS_ROLE_WS`\nand `LWS_ROLE_MQTT` enabled so it can handle any kind of policy.\n\n","s":{"c":1748971573,"u": 390}} ],"g": 9932,"chitpc": 0,"ehitpc": 0,"indexed":0 , "ab": 1, "si": 0, "db":0, "di":1, "sat":0, "lfc": "0000"}