libwebsockets
Lightweight C library for HTML5 websockets
Loading...
Searching...
No Matches
Notes about lwsws

Libwebsockets Web Server

lwsws is an implementation of a very lightweight, ws-capable generic web server, which uses libwebsockets to implement everything underneath.

If you are basically implementing a standalone server with lws, you can avoid reinventing the wheel and use a debugged server including lws, managed mainly by JSON. You can offer most things using the mounts in JSON, but for custom services like ws protocols, you do this in code and then tell lwsws to offer the protocol on the vhost(s) desired.

Build

Just enable -DLWS_WITH_LWSWS=1 at cmake-time.

It enables libuv and plugin support automatically.

Special user for lwsws

You can specify a user and group for lwsws to run as in the global section of the config file. Here, I use user 48 ("apache") to run lwsws itself. You can create such a user with sudo useradd -u 48 -M -r apache.

Lwsws Configuration

lwsws uses JSON config files, they're pure JSON except:

  • '#' may be used to turn the rest of the line into a comment.
  • There's also a single substitution, if a string contains "_lws_ddir_", then that is replaced with the LWS install data directory path, eg, "/usr/share" or whatever was set when LWS was built + installed. That lets you refer to installed paths without having to change the config if your install path was different.
  • Scoped substitution defines: a pair whose name starts with '=' is not part of the config schema, it defines a substitution symbol instead. See "Scoped substitution defines" below.

There is a single file intended for global settings. If lwsws starts as root, it switches to the user and group in "username" / "groupname" (names) or "uid" / "gid" (each either a numeric id or a name) once it is listening; a value that is not a numeric id or a known name is a config error. If the switch fails, lwsws exits rather than serve as root. See "Dropping root privileges" in README.coding.md.

/etc/lwsws/conf

# these are the server global settings
# stuff related to vhosts should go in one
# file per vhost in ../conf.d/
{
"global": {
"username": "apache",
"groupname": "apache",
"count-threads": "1",
"server-string": "myserver v1", # returned in http headers
"ws-pingpong-secs": "200", # confirm idle established ws connections this often
"init-ssl": "yes"
}
}

and a config directory intended to take one file per vhost

/etc/lwsws/conf.d/warmcat.com

{
"vhosts": [{
"name": "warmcat.com",
"port": "443",
"interface": "eth0", # optional
"host-ssl-key": "/etc/pki/tls/private/warmcat.com.key", # if given enable ssl
"host-ssl-cert": "/etc/pki/tls/certs/warmcat.com.crt",
"host-ssl-ca": "/etc/pki/tls/certs/warmcat.com.cer",
"mounts": [{ # autoserve
"mountpoint": "/",
"origin": "file:///var/www/warmcat.com",
"default": "index.html"
}]
}]
}

To get started quickly, an example config reproducing the old test server on port 7681, non-SSL is provided. To set it up

# mkdir -p /etc/lwsws/conf.d /var/log/lwsws
# cp ./lwsws/etc-lwsws-conf-EXAMPLE /etc/lwsws/conf
# cp ./lwsws/etc-lwsws-conf.d-localhost-EXAMPLE /etc/lwsws/conf.d/test-server
# sudo lwsws

Scoped substitution defines

A pair whose name starts with = is not passed through to the config schema. It defines a substitution symbol, and its value is used to expand ${NAME} sequences in string values of the config that are produced while it is in scope:

{
"global": {
"=PKI_ROOT": "/var/dnssec", # define PKI_ROOT for this object
"server-string": "lwsws",
"reject-service-keywords": [{
"=BADBOT": "scumbot/1.0", # define scoped to this array item
"name": "${BADBOT}" # -> "scumbot/1.0"
}]
}
}

The rules are:

  • A define is visible from where it is defined, until the close of the JSON object it was defined in. The value string may itself use ${NAME} sequences referencing defines already defined in the same or an enclosing object.
  • Defines in an inner object may shadow ones with the same name from enclosing objects. Within the same object, the redefinition replaces the earlier define from that point on.
  • Defines in the "global" section of the globals config files (conf and conf.d/) are install-wide: they accumulate across the globals config fragments, and stay visible in all the vhost config files, until all config files have been walked. Defines in the vhost config files themselves stay isolated to the file they were created in.
  • Each file is otherwise parsed with a fresh set of defines, so defines do not leak between unrelated files.
  • Define names are 1...31 characters from a-z, A-Z, 0-9 and _.
  • Strings are substituted as they are parsed, so any string value in the config can use ${NAME}, including values used to build other define values. Substitution applies to string values only, pair names are not substituted.
  • Problems fail the config load loudly at the file and line concerned: using an undefined symbol, an invalid define name, or giving a define a non-string value. Pair names starting with = are reserved for this.

Using Letsencrypt or other ACME providers

Lws supports automatic provisioning and renewal of TLS certificates.

See ./READMEs/README.plugin-acme.md for examples of how to set it up on an lwsws vhost.

Other Global Options

  • reject-service-keywords allows you to return an HTTP error code and message of your choice if a keyword is found in the user agent
"reject-service-keywords": [{
"scumbot": "404 Not Found"
}]
  • timeout-secs lets you set the global timeout for various network-related operations in lws, in seconds. It defaults to 5.
  • http-header-data sets the maximum bytes of http header lws will buffer per connection, for server and client connections. It defaults to 4096; sites that deal with very fat response headers (for example, release redirects from github with many cookies) should raise it, eg "8192". The maximum is 32768.
  • unproven-limit-wsi (with LWS_WITH_PEER_LIMITS) sets how many connections whose peer address is not proven yet, eg, QUIC connections whose address QUIC has not validated yet, may exist at once, all together, since their address is whatever the sender wrote; they are not counted against that address's ip-limit-wsi until it is proven. It defaults to 0, no limit. Set it to the number of such connections you expect at one time, with room to spare: a new one beyond it is dropped, while connections already proven are not affected. But a nonzero limit trades availability for bounded unproven state: an unvalidated QUIC connection lives until its handshake times out (timeout-secs), and a sender spoofing source addresses needs no answer, so about unproven-limit-wsi / timeout-secs Initials a second keep the budget full, and every new QUIC client is refused for as long as that goes on. That is why it defaults to 0: unvalidated QUIC connections are still bounded by the half-open limit, which makes room for a new one by evicting the oldest, so real clients still get in.
  • http-header-pool-per-mux sets how many request header tables the streams of one h2 or h3 connection may hold at once while they wait on the client, eg, for their bodies. A new stream with more to come over the limit is refused on its own (h2 REFUSED_STREAM, h3 H3_REQUEST_REJECTED, both safe for the client to retry), so a few connections opening many streams that never finish their bodies cannot use up the pool everybody's requests need. Requests that arrive whole, like a burst of GETs, are never refused for it. The default is one less than the pool's tables per fd, but at least 8: with the default pool that is 8.

Lwsws Vhosts

One server can run many vhosts, where SSL is in use SNI is used to match the connection to a vhost and its vhost-specific SSL keys during SSL negotiation.

Listing multiple vhosts looks something like this

{
"vhosts": [ {
"name": "localhost",
"port": "443",
"host-ssl-key": "/etc/pki/tls/private/libwebsockets.org.key",
"host-ssl-cert": "/etc/pki/tls/certs/libwebsockets.org.crt",
"host-ssl-ca": "/etc/pki/tls/certs/libwebsockets.org.cer",
"mounts": [{
"mountpoint": "/",
"origin": "file:///var/www/libwebsockets.org",
"default": "index.html"
}, {
"mountpoint": "/testserver",
"origin": "file:///usr/local/share/libwebsockets-test-server",
"default": "test.html"
}],
# which protocols are enabled for this vhost, and optional
# vhost-specific config options for the protocol
#
"ws-protocols": [{
"warmcat,timezoom": {
"status": "ok"
}
}]
},
{
"name": "localhost",
"port": "7681",
"host-ssl-key": "/etc/pki/tls/private/libwebsockets.org.key",
"host-ssl-cert": "/etc/pki/tls/certs/libwebsockets.org.crt",
"host-ssl-ca": "/etc/pki/tls/certs/libwebsockets.org.cer",
"mounts": [{
"mountpoint": "/",
"origin": ">https://localhost"
}]
},
{
"name": "localhost",
"port": "80",
"mounts": [{
"mountpoint": "/",
"origin": ">https://localhost"
}]
}
]
}

That sets up three vhosts all called "localhost" on ports 443 and 7681 with SSL, and port 80 without SSL but with a forced redirect to https://localhost

Vhost name and port sharing

The vhost name field is used to match on incoming SNI or Host: header, so it must always be the host name used to reach the vhost externally.

  • Vhosts may have the same name and different ports, these will each create a listening socket on the appropriate port.
  • Vhosts may also have the same port and different name: these will be treated as true vhosts on one listening socket and the active vhost decided at SSL negotiation time (via SNI) or if no SSL, then after the Host: header from the client has been parsed.
  • Only the vhosts on the listening socket a connection arrived on can be chosen for it, by SNI, Host:, :authority or a tls passthrough. Vhosts on the same port but a different interface have listening sockets of their own, so a peer reaching one interface cannot name his way onto a vhost that listens only on another, eg, an admin vhost on lo beside a public one on the same port. An unknown name falls back only among the vhosts on his listening socket too.
  • A tls connection whose SNI name matches no vhost on the listener it arrived on is refused, with a fatal unrecognized_name alert. Otherwise the certificate the peer is shown, and the client certificate policy he meets, would be decided by vhost creation order just because he named something that does not exist. A vhost may be nominated to take those connections instead with "sni-fallback": "1", see below.
  • A connection that sends no SNI at all is not affected by that, it stays on the vhost that accepted it; neither is selection by Host: header, since a server with a single vhost is reached by IP address all the time.
  • All the tls backends do this the same way. OpenSSL, mbedtls, gnutls and openHiTLS act on it in their servername callback; BearSSL and Schannel have no callback that can still change the certificate at that point (Schannel has none at all, and BearSSL's chain-selection hook runs after ALPN and the cipher suites are already settled), so on those two lws reads the name out of the ClientHello itself, before handing the first record to the tls library. The only case those two treat differently is a ClientHello split across several tls records, which no client does: lws does not reassemble it, and such a peer is served as if he had sent no SNI.

Lwsws Protocols

Vhosts by default have available the union of any initial protocols from context creation time, and any protocols exposed by plugins.

Vhosts can select which plugins they want to offer and give them per-vhost settings using this syntax

"ws-protocols": [{
"warmcat-timezoom": {
"status": "ok"
}
}]

The "x":"y" parameters like "status":"ok" are made available to the protocol during its per-vhost LWS_CALLBACK_PROTOCOL_INIT (in is a pointer to a linked list of struct lws_protocol_vhost_options containing the name and value pointers).

To indicate that a protocol should be used when no Protocol: header is sent by the client, you can use "default": "1"

"ws-protocols": [{
"warmcat-timezoom": {
"status": "ok",
"default": "1"
}
}]

Similarly, if your vhost is serving a raw protocol, you can mark the protocol to be selected using "raw": "1"

"ws-protocols": [{
"warmcat-timezoom": {
"status": "ok",
"raw": "1"
}
}]

See also "apply-listen-accept" below.

Care about redirecting the trailing /

In lws the mount applies at the URL folder, so a mount at /myapp actually works starting from /myapp/.

This means you often need to add an additional redirect mount to do the last step for the user who may have typed .../myapp, to get them to .../myapp/ where the app actually takes over. 

    "mountpoint": "/git",
    "origin": ">https://warmcat.com/git/",

Lwsws Other vhost options

  • If the three options host-ssl-cert, host-ssl-ca and host-ssl-key are given, then the vhost supports SSL.

Each vhost may have its own certs, SNI is used during the initial connection negotiation to figure out which certs to use by the server name it's asking for from the request DNS name.

When the cert and key files are renewed on disk, eg, by moving a -latest symlink onto new files the way the acme plugin, the cert distribution client or certbot do, the vhost notices and moves onto them without a restart and without disturbing its connections. Connections made before the renewal carry on with the old cert until they close. The files must still be readable by lwsws after it has dropped its privileges for it to see the renewal by itself; a warning is logged once if they aren't. A renewer that keeps them out of lwsws's reach, like the cert distribution client, hands the vhost copies of the new pair instead (lws_tls_cert_updated()).

  • "`host-ssl-cert-grace-secs`": "900" holds a renewed cert back for that many seconds after it was issued or written, whichever is later, before the vhost serves it. The renewed pair is loaded as soon as it is seen, so it is served when the time is up even if it can't be read by then; a restart serves it at once. Use it when the cert is published in DNS for DANE, so the new TLSA record has replaced cached copies of the old one first. Default 0, the renewed cert is served as soon as it is seen.
  • "`http-body-min-rate`": "500" is the slowest, in bytes per second on average, that a request body may arrive on this vhost, after its grace time "`http-body-grace-secs`": "20" from the request's headers. The body is allowed the grace time plus a second for every http-body-min-rate bytes of it so far; one falling behind is given up on as for a timeout (an h2 or h3 stream is reset, an h1 connection closed), so a client cannot hold a request open for ever by trickling its body. Those are the defaults, which cut no upload faster than 4 kbit/s whatever its size. Time the server itself is not reading the body, eg, while a cgi or a proxy mount's onward connection is slow to take it, counts as well. "none" leaves only the timeout between pieces of body (timeout-secs).
  • "`http-response-stall-secs`": "300" is how long a response on this vhost may go without any more of it being sent, eg, because the peer stopped reading it, before it is given up on. Each piece sent starts it again, so it bounds a stall, not how long the whole response takes. A browser saving a big download to a slow device can stop reading for minutes while it drains its cache; on a proxy, set it on the vhost the browser talks to as well as on the one being proxied to. Default 0, timeout-secs but at least 30s.
  • "`mux-reset-limit`": "200" is the most streams of one h2 or h3 connection that may end before they are answered, on average per "`mux-reset-window-secs`": "10". The client resetting a stream it had not had its answer for counts, and so does the server resetting one for any reason but having answered it (eg, its body coming too slowly), other than refusing it for want of a header table. Going over it ends the connection (GOAWAY ENHANCE_YOUR_CALM on h2, H3_EXCESSIVE_LOAD on h3), so a client cannot have requests started for it as fast as it can send them by resetting each one after it went out ("rapid reset"). Those are the defaults; "none" means no limit.
  • "`sni-fallback`": "1" makes this vhost the one that serves tls connections arriving on its listen port with an SNI name that matches no vhost there. At most one vhost per port needs it; if none has it, such connections are refused with a fatal unrecognized_name alert and one rate-limited notice-level log line. Give it to the vhost whose certificate and client-certificate policy you are happy for an unknown name to meet, typically the "main" vhost on the port. Connections that send no SNI at all never come here, they stay on the vhost that accepted them.
  • "`no-http-stream-compression`": "1" stops the vhost compressing its http responses on the fly, whatever the client's Accept-Encoding says. Use it where a response mixes secrets with content the client influences, since the compressed length can leak the secret (BREACH). Without it, in a build with LWS_WITH_HTTP_STREAM_COMPRESSION, text-ish responses are compressed when the client accepts it, and say vary: accept-encoding so caches keep the codings apart. A plugin can instead keep just one response uncompressed by calling lws_http_compression_opt_out() before composing its headers; the lws-login and auth-server plugins do that for their pages.
  • keepalive_timeout (in secs, default 5) is how long the vhost waits on a client connection with nothing outstanding on it before closing it. A mount can override it for requests to that mount with keepalive-timeout.
  • interface lets you specify which network interface to listen on, if not given listens on all. If the network interface is not usable (eg, ethernet cable out) it will be logged at startup with such vhost not listening, and lws will poll for it and bind a listen socket to the interface if and when it becomes available.
  • "`unix-socket`": "1" causes the unix socket specified in the interface option to be used instead of an INET socket
  • "`unix-socket-perms`": "user:group" allows you to control the unix permissons on the listening unix socket. It's always get to 0600 mode, but you can control the user and group for the socket fd at creation time. This allows you to use unix user and groups to control who may open the other end of the unix socket on the local system.
  • "`sts`": "1" causes lwsws to send a Strict Transport Security header with responses that informs the client he should never accept to connect to this address using http. This is needed to get the A+ security rating from SSL Labs for your server.

  • "`access-log`": "filepath" sets where apache-compatible access logs will be written

    Each request is always one line. Quotes in the request, referrer and user agent are logged as ' and control characters as _, and when a line would be too long, the referrer and user agent are cut short, never the quotes or the newline.

    The first field of each line is the client's address, eg 192.0.2.10, exactly as it always was, when that is all lws knows of who the peer is and the connection's handshake proved it (for h3, QUIC's handshake on a validated path); a client on an AF_UNIX socket is unix. When the connection has more layers than its own peer's address, eg the client a trusted front named in a PROXY v2 header (see proxy-v2-trusted), or its only one is not proven, eg the source address of datagrams, the field is every layer instead, still as one field with no spaces: top (the most specific claim) first, joined by <, each address[:port]~cred, where cred is the initial of how far the layer can be believed, hearsay, vouched, routable or crypto, and an IPv6 address with a port is in brackets, eg

    192.0.2.10:40000~v<10.0.0.2:40122~r - - [19/Oct/2026:12:00:00 +0000] "GET / HTTP/1.1" 200 0 "" ""

    is a client 192.0.2.10 that the front at 10.0.0.2, whose own connection to us completed its TCP handshake, vouched for. A parser that wants a single address can take the field up to the first ~: that is the most specific claim of who the client is, believable as far as its cred says. See lws_peer_entries_describe() in lws-network-helper.h for the format.

  • "enable-client-ssl": "1" enables the vhost's client SSL context, you will need this if you plan to create client conections on the vhost that will use SSL. You don't need it if you only want http / ws client connections.
  • "`ciphers`": "<cipher list>" sets the allowed list of TLS <= 1.2 ciphers and key exchange protocols for the serving SSL_CTX on the vhost. With OPENSSL, the default list is restricted to only those providing PFS (Perfect Forward Secrecy) on the author's Fedora system.

With GNUTLS, this string is used verbatim as the gnutls priority string for the vhost's server side, and controls the protocol versions as well as the ciphers. The default is "`NORMAL:-VERS-TLS1.0:-VERS-TLS1.1`", ie, the gnutls NORMAL set with TLS1.0 and TLS1.1 disabled, since offering those caps the server at a "B" on ssllabs.com. If you need TLS1.2 only, use eg "`NORMAL:-VERS-ALL:+VERS-TLS1.2:+VERS-TLS1.3`".

If you need to allow weaker ciphers, you can provide an alternative list here per-vhost.

  • "`client-ssl-ciphers`": "<cipher list>" sets the allowed list of <= TLS1.2 ciphers and key exchange protocols for the client SSL_CTX on the vhost. With GNUTLS, this is the gnutls priority string for the vhost's client side, defaulting to "`NORMAL:-VERS-TLS1.0:-VERS-TLS1.1`".
  • "`tls13-ciphers`": "<cipher list>" OPENSSL 1.1.1+ only: sets allowed list of TLS1.3+ ciphers and key exchange protocols for the client SSL_CTX on the vhost. The default is to allow all.
  • "`client-tls13-ciphers`": "<cipher list>" OPENSSL 1.1.1+ only: sets the allowed list of TLS1.3+ ciphers and key exchange protocols for the client SSL_CTX on the vhost. The default is to allow all.
  • "`ecdh-curve`": "<group list>" OPENSSL only: colon-separated list of TLS group names (eg, "X25519:P-384", or a single group name like "prime256v1") restricting the groups used for key exchange on the vhost's serving SSL_CTX. On old OpenSSL (<1.1.0), only a single curve name is understood and the default is "prime256v1".
  • "`client-ecdh-curve`": "<group list>" OPENSSL only: colon-separated list of TLS group names restricting the groups offered in the ClientHello by the vhost's client SSL_CTX. If not given, ecdh-curve is used for the client SSL_CTX too. This is useful to stop clients offering groups some servers cannot cope with, for example the post-quantum and ffdhe groups that OpenSSL 3.4+ includes in its defaults.
  • "`noipv6`": "on" Disable ipv6 completely for this vhost
  • "`ipv6only`": "on" Only allow ipv6 on this vhost / "off" only allow ipv4 on this vhost
  • "`ssl-option-set`": "<decimal>" Sets the SSL option flag value for the vhost. It may be used multiple times and OR's the flags together.

The values are derived from /usr/include/openssl/ssl.h

# define SSL_OP_NO_TLSv1_1 0x10000000L

would equate to

"`ssl-option-set`": "268435456"
  • "`ssl-option-clear'": "<decimal>" Clears the SSL option flag value for the vhost. It may be used multiple times and OR's the flags together.
  • "`ssl-client-option-set`" and "`ssl-client-option-clear`" work the same way for the vhost Client SSL context
  • "`headers':: [{ "header1": "h1value", "header2": "h2value" }] allows you to set arbitrary headers on every file served by the vhost recommended vhost headers for good client security are @icode "headers": [{ "Content-Security-Policy": "script-src 'self'", "X-Content-Type-Options": "nosniff", "X-XSS-Protection": "1; mode=block", "X-Frame-Options": "SAMEORIGIN" }] @endicode - "`apply-listen-accept`": "on" This vhost only serves a non-http protocol, specified in "listen-accept-role" and "listen-accept-protocol" @section autotoc_md532 Lwsws Mounts Where mounts are given in the vhost definition, then directory contents may be auto-served if it matches the mountpoint. Mount protocols are used to control what kind of translation happens - file:// serve the uri using the remainder of the url past the mountpoint based on the origin directory. Eg, with this mountpoint @icode { "mountpoint": "/", "origin": "file:///var/www/mysite.com", "default": "/" } @endicode The uri /file.jpg would serve /var/www/mysite.com/file.jpg, since / matched. - ^http:// or ^https:// these cause any url matching the mountpoint to issue a redirect to the origin url - cgi:// this causes any matching url to be given to the named cgi, eg @icode { "mountpoint": "/git", "origin": "cgi:///var/www/cgi-bin/cgit", "default": "/" }, { "mountpoint": "/cgit-data", "origin": "file:///usr/share/cgit", "default": "/" }, @endicode would cause the url /git/myrepo to pass "myrepo" to the cgi /var/www/cgi-bin/cgit and send the results to the client. - http:// or https:// these perform reverse proxying, serving the remote origin content from the mountpoint. Eg @icode { "mountpoint": "/proxytest", "origin": "https://libwebsockets.org" } @endicode This will cause your local url <tt>/proxytest</tt> to serve content fetched from libwebsockets.org over ssl; whether it's served from your server using ssl is unrelated and depends how you configured your local server. Notice if you will use the proxying feature, <tt>LWS_WITH_HTTP_PROXY</tt> is required to be enabled at cmake, and for <tt>https</tt> proxy origins, your lwsws configuration must include <tt>\"init-ssl\": \"1\"</tt> and the vhost with the proxy mount must have <tt>\"enable-client-ssl\": \"1\"</tt>, even if you are not using ssl to serve. <tt>/proxytest/abc</tt>, or <tt>/proxytest/abc?def=ghi</tt> etc map to the origin + the part past <tt>/proxytest</tt>, so links and img src urls etc work as do all urls under the origin path. In addition link and src urls in the document are rewritten so / or the origin url part are rewritten to the mountpoint part. @subsection autotoc_md533 Lwsws Other mount options 1) Some protocols may want "per-mount options" in name:value format. You can provide them using "pmo" @iverbatim { "mountpoint": "/stuff", "origin": "callback://myprotocol", "pmo": [{ "myname": "myvalue" }] }

Note on PMO vs. PVO: Per-vhost options (PVOs) are provided to the protocol at LWS_CALLBACK_PROTOCOL_INIT time and are applied universally once per vhost. Per-mount options (PMOs), on the other hand, are strictly tied to specific mounts and are accessed dynamically during connection and request handling (such as inside LWS_CALLBACK_HTTP). Crucially, a protocol must explicitly choose to query the lws_http_mount structure using lws_pmo_search or lws_pmo_get_str to look for its specific PMO names in order to respect them, bypassing its globally cached PVO equivalents.

2) When using a cgi:// protocol origin at a mountpoint, you may also give cgi environment variables specific to the mountpoint like this

{
"mountpoint": "/git",
"origin": "cgi:///var/www/cgi-bin/cgit",
"default": "/",
"cgi-env": [{
"CGIT_CONFIG": "/etc/cgitrc/libwebsockets.org"
}]
}

This allows you to customize one cgi depending on the mountpoint (and / or vhost).

3) It's also possible to set the cgi timeout (in secs) per cgi:// mount, like this

"cgi-timeout": "30"

4) callback:// protocol may be used when defining a mount to associate a named protocol callback with the URL namespace area. For example

{
"mountpoint": "/formtest",
"origin": "callback://protocol-post-demo"
}

All handling of client access to /formtest[anything] will be passed to the callback registered to the protocol "protocol-post-demo".

This is useful for handling POST http body content or general non-cgi http payload generation inside a plugin.

See the related notes in README.coding.md

connect:// is how a vhost accepts an h1 CONNECT: the mountpoint is the authority asked for, as host:port, and the connection becomes raw, bound to the named protocol. A CONNECT nothing is mounted for is refused with a 405.

{
"mountpoint": "tunnel.example.com:443",
"origin": "connect://my-raw-protocol"
}

See README.http-fallback.md for how the protocol keeps the connection valid.

5) Cache policy of the files in the mount can also be set. If no options are given, the content is marked uncacheable.

{
"mountpoint": "/",
"origin": "file:///var/www/mysite.com",
"cache-max-age": "60", # seconds
"cache-reuse": "1", # allow reuse at client at all
"cache-revalidate": "1", # check it with server each time
"cache-intermediaries": "1" # allow intermediary caches to hold
}

6) You can also define a list of additional mimetypes per-mount

"extra-mimetypes": {
".zip": "application/zip",
".doc": "text/evil"
}

Normally a file suffix MUST match one of the canned mimetypes or one of the extra mimetypes, or the file is not served. This adds a little bit of security because even if there is a bug somewhere and the mount dirs are circumvented, lws will not serve, eg, /etc/passwd.

If you provide an extra mimetype entry 

                    "*": ""

Then any file is served, if the mimetype was not known then it is served without a Content-Type: header.

7) A mount can be protected by HTTP Basic Auth. This only makes sense when using https, since otherwise the password can be sniffed.

You can add a basic-auth entry on an http mount like this

{
"mountpoint": "/basic-auth",
"origin": "file://_lws_ddir_/libwebsockets-test-server/private",
"basic-auth": "/var/www/balogins-private"
}

Before serving anything, lws will signal to the browser that a username / password combination is required, and it will pop up a dialog. When the user has filled it in, lwsws checks the user:password string against the text file named in the basic-auth entry.

The file should contain user:pass one per line

testuser:testpass
myuser:hispass

The file should be readable by lwsws, and for a little bit of extra security not have a file suffix, so lws would reject to serve it even if it could find it on a mount.

After successful authentication, lws_http_basic_auth_user(wsi) returns the authenticated username until the transaction completes, whether or not the request headers are still around. WSI_TOKEN_HTTP_AUTHORIZATION is also rewritten to contain it, while the headers last.

The mount's basic auth, like its interceptors, also gates a ws upgrade (on h1, or ws over h2 / h3) whose path lands in the mount, before it is bound to a ws protocol or, on a reverse proxy mount, relayed to the backend. Without valid credentials it is refused with 401 and the WWW-Authenticate challenge, as an http request to the mount would be.

8) You can also control the exact path matching and redirect behavior per-mount.

{
"mountpoint": "/",
"origin": ">https://warmcat.com",
"exact-match": "1",
"append-path": "1"
}
  • "exact-match": "1" forces the mount to only match if the request URL is exactly the mountpoint (no directory prefix matching).
  • "append-path": "1" can be used with HTTP/HTTPS redirects (>http:// or >https://). By default, redirects strictly redirect to the origin without appending the trailing path of the URL. Setting this flag will append the remainder of the request URL to the redirect destination, always as a path after the origin: a / is put between them if neither has one.
  • "no-ws-upgrades": "1" instructs the router to ignore this mount if the incoming HTTP request is a WebSocket upgrade request (contains an Upgrade: header). This is useful to prevent WSS connections from being accidentally swallowed by a broad alias or redirect mount (e.g. /) when they were intended for a different, overlapping protocol.

In the case you want to also protect being able to connect to a ws protocol on a particular vhost by requiring the http part can authenticate using Basic Auth before the ws upgrade, this is also possible. In this case, the "basic-auth": and filepath to the credentials file is passed as a pvo in the "ws-protocols" section of the vhost definition.

Using mount interception

The mounts in lws allow you to stack up other plugins that run "before" the main mountpoint. There are two "interceptor plugins" provided which can be useful for this, lws_login and lws_captcha_ratelimit, you can create your own based on these.

To indicate you want to use an interceptor plugin for a mount, you add an "interceptor-path" entry to the original mount definition, pointing to the mountpoint of the interceptor plugin, like this

{
"mountpoint": "/",
"origin": "file:///var/www/mysite.com",
"interceptor-path": "/lws-login"
},
{
"mountpoint": "/lws-login",
"origin": "callback://lws-login"
}

With this arrangement, the original mountpoint will only be visible once the intercepting protocol is satisfied, either by a correct login for the login one, or by the captcha / ratelimit one being satisfied.

Requiring a Client Cert on a vhost

You can make a vhost insist to get a client certificate from the peer before allowing the connection with

"client-cert-required": "1"

the connection will only proceed if the client certificate was signed by the same CA as the server has been told to trust.

Configuring Fallback and Raw vhosts

Lws supports some unusual modes for vhost listen sockets, which may be configured entirely using the JSON per-vhost config language in the related vhost configuration section.

There are three main uses for them

1) A vhost bound to a specific role and protocol, not http. This binds all incoming connections on the vhost listen socket to the "raw-proxy" role and protocol "myprotocol".

"listen-accept-role": "raw-proxy",
"listen-accept-protocol": "myprotocol",
"apply-listen-accept": "1"

2) A vhost that wants to treat noncompliant connections for http or https as belonging to a secondary fallback role and protocol. This causes non-https connections to an https listener to stop being treated as https, to lose the tls wrapper, and bind to role "raw-proxy" and protocol "myprotocol". For example, connect a browser on your external IP :443 as usual and it serves as normal, but if you have configured the raw-proxy to portforward 127.0.0.1:22, then connecting your ssh client to your external port 443 will instead proxy your sshd over :443 with no http or tls getting in the way.

"listen-accept-role": "raw-proxy",
"listen-accept-protocol": "myprotocol",
"fallback-listen-accept": "1",
"allow-non-tls": "1"

3) A vhost wants to either redirect stray http traffic back to https, or to actually serve http on an https listen socket (this is not recommended since it allows anyone to drop the security assurances of https by accident or design).

"allow-non-tls": "1",
"redirect-http": "1",

...or,

"allow-non-tls": "1",
"allow-http-on-https": "1",

These are read from the vhost that owns the listen socket, the first one created on that interface and port. The plaintext connections they let in are never served by another vhost sharing the listener: one whose Host: names another vhost is refused with a 421, or with redirect-http only redirected to the same url over https.

Passing a vhost's tls through to another server

A vhost can have its tls connections passed, still encrypted, to another server that terminates the tls itself, eg, a service on another host that keeps its own private key. It is chosen by the SNI name in the ClientHello, like any vhost on a shared listener, and needs no certificate, mounts or protocols of its own. This needs lws built with LWS_WITH_TLS_PASSTHROUGH.

{
"name": "auth.example.com",
"port": "443",
"tls-passthrough": "192.168.1.20:443",
"tls-passthrough-proxy-v2": "1"
}

The connection's first bytes are only looked at, never consumed, so a connection naming any other vhost goes on to the listener's own tls exactly as before. The ClientHello may arrive in several tcp segments, or be split across several tls records; up to 32KB of it is gathered to find the name. Make the vhosts that terminate tls on the port before any passed through one, so the listener is theirs; on a listener with only passed through vhosts, anything not for one of them is dropped.

The server only sees the connection come from us, so with "tls-passthrough-proxy-v2": "1" we send it a PROXY v2 header first, as haproxy defined it, naming the real client and the SNI name it asked for. The server must expect it: an lws server is told to with "proxy-v2-trusted", below.

QUIC (h3)

When the passthrough target is a numeric address and "tls-passthrough-proxy-v2" is on, the vhost's QUIC is passed through too, to the same address and port over udp. The front reads the SNI name from the client's first Initial packets, which anyone can decrypt (RFC 9001 5.2), without terminating anything; a connection for any other name goes on to the front's own QUIC as before. Each datagram is relayed with a PROXY v2 header naming the client, and the server's datagrams back must carry one naming the client they are for, so the server's QUIC sees each client's real address. When a client moves, eg, a NAT rebinding gives it a new port, the front follows its flow by connection ID, and the server validates the new path itself; the front sends a client only to an address the server names that the flow has seen it at.

The server behind it must list the front in "proxy-v2-trusted", which applies to its udp listener as much as its tcp one. A target given as a name is passed through for tcp only. The headers add up to 52 bytes to each datagram between the front and the server, so that path wants an MTU a little above the clients'. The front routes the server's packets by the connection IDs it sees in their long headers, which is all an lws server uses; a server that hands out more with NEW_CONNECTION_ID frames, which are encrypted, can't have a client that moves onto one of those followed.

Accepting PROXY v2 headers

A vhost that is reached through a proxy that doesn't terminate the connection, eg, the passthrough above or haproxy, can be told which proxies send a PROXY v2 header ahead of what they forward:

"proxy-v2-trusted": "192.168.1.1, 2001:db8::/64"

It is a comma-separated list of numeric addresses or CIDR netblocks. A connection from one of them must start with exactly one valid PROXY v2 header, or it is dropped; the header is taken off the connection, and the client it names is the peer address lws reports from then on, eg, to a plugin keying rate limits on it. A connection from anywhere else is never looked at for one, so nobody else can claim to be somebody they are not, and a header after the first, wherever it came from, is just the client's data. The list applies to every vhost on the same listener.

That is as good as the source address it rests on. A TCP handshake proves it; a datagram's source is whatever its sender wrote, port included, so on a QUIC listener the client a datagram's header names, and the relay it came from, are only hearsay until the client's QUIC connection validates its address, which our packets can only do by reaching it through that relay. Logs and the access log show such a client as a relay claim above the relay, each with its cred. Once proven, a relay is never moved or pushed out by datagrams from anywhere else, which still reach QUIC under the claim, where they are dropped unless they decrypt; before that, a sender who can spoof the front's address can disturb a client's handshake, or make the server keep some unproven state, as anyone sending Initials can. So keep the path from the front to the server one its address can't be spoofed on, eg, loopback or a private link.

The global per-IP limits, "ip-limit-wsi" and "ip-limit-ah", count a proxied connection against the client its header names, not the proxy, so one client behind the proxy can't spend the budget of all the others; a connection the proxy makes for itself, with a LOCAL header, counts against the proxy. Until its header has arrived, a connection from a trusted proxy counts against nobody: it is bounded by the accept timeout and the fd limit instead.

Lwsws Plugins

Protcols and extensions may also be provided from "plugins", these are lightweight dynamic libraries. They are scanned for at init time, and any protocols and extensions found are added to the list given at context creation time.

Protocols receive init (LWS_CALLBACK_PROTOCOL_INIT) and destruction (LWS_CALLBACK_PROTOCOL_DESTROY) callbacks per-vhost, and there are arrangements they can make per-vhost allocations and get hold of the correct pointer from the wsi at the callback.

This allows a protocol to choose to strictly segregate data on a per-vhost basis, and also allows the plugin to handle its own initialization and context storage.

To help that happen conveniently, there are some new apis

  • lws_vhost_get(wsi)
  • lws_protocol_get(wsi)
  • lws_callback_on_writable_all_protocol_vhost(vhost, protocol)
  • lws_protocol_vh_priv_zalloc(vhost, protocol, size)
  • lws_protocol_vh_priv_get(vhost, protocol)

dumb increment, mirror and status protocol plugins are provided as examples.

Additional plugin search paths

Packages that have their own lws plugins can install them in their own preferred dir and ask lwsws to scan there by using a config fragment like this, in its own conf.d/ file managed by the other package

{
"global": {
"plugin-dir": "/usr/local/share/coherent-timeline/plugins"
}
}

lws-server-status plugin

One provided protocol can be used to monitor the server status.

Enable the protocol like this on a vhost's ws-protocols section

"lws-server-status": {
"status": "ok",
"update-ms": "5000"
}

"update-ms" is used to control how often updated JSON is sent on a ws link.

And map the provided HTML into the vhost in the mounts section

{
"mountpoint": "/server-status",
"origin": "file:///usr/local/share/libwebsockets-test-server/server-status",
"default": "server-status.html"
}

You might choose to put it on its own vhost which has "interface": "lo", so it's not externally visible, or use the Basic Auth support to require authentication to access it.

"hide-vhosts": "{0 | 1}" lets you control if information about your vhosts is included. Since this includes mounts, you might not want to leak that information, mount names, etc.

"filespath":"{path}" lets you give a server filepath which is read and sent to the browser on each refresh. For example, you can provide server temperature information on most Linux systems by giving an appropriate path down /sys.

This may be given multiple times.

Lwsws Configuration Reload

You may send lwsws a HUP signal, by, eg

$ sudo systemctl reload lwsws

or by sending it to the specific lwsws PID kill -HUP <pid>

This causes lwsws to "deprecate" the existing lwsws process, and remove and close all of its listen sockets, but otherwise allowing it to continue to run, until all of its open connections close.

When a deprecated lwsws process has no open connections left, it is destroyed automatically.

After sending the SIGHUP to the main lwsws process, a new lwsws process, which can pick up the newly-available listen sockets, and use the current configuration files, is automatically started.

The new configuration may differ from the original one in arbitrary ways, the new context is created from scratch each time without reference to the original one.

Notes

1) Protocols that provide a "shared world" like mirror will have as many "worlds" as there are lwsws processes still active. People connected to a deprecated lwsws process remain connected to the existing peers.

But any new connections will apply to the new lwsws process, which does not share per-vhost "shared world" data with the deprecated process. That means no new connections on the deprecated context, ie a "shrinking world" for those guys, and a "growing world" for people who connect after the SIGHUP.

2) The new lwsws process owes nothing to the previous one. It starts with fresh plugins, fresh configuration, fresh root privileges if that how you start it.

The plugins may have been updated in arbitrary ways including struct size changes etc, and lwsws or lws may also have been updated arbitrarily.

3) A root parent process is left up that is not able to do anything except respond to SIGHUP or SIGTERM. Actual serving and network listening etc happens in child processes which use the privileges set in the lwsws config files.

Lwsws Integration with Systemd

lwsws needs a service file like this as /usr/lib/systemd/system/lwsws.Built-in service loop entry

[Unit]
Description=Libwebsockets Web Server
After=syslog.target
[Service]
ExecStart=/usr/local/bin/lwsws
ExecReload=/usr/bin/kill -HUP $MAINPID
[Install]
WantedBy=multi-user.target

You can find this prepared in ./lwsws/usr-lib-systemd-system-lwsws.Built-in service loop entry

Note that lwsws natively handles SIGTERM, which systemd sends by default when stopping a service. This enables graceful isolated shutdown of individual lwsws configurations in their own control groups without requiring any manual ExecStop commands.

Lwsws Integration with logrotate

For correct operation with logrotate, /etc/logrotate.d/lwsws (if that's where we're putting the logs) should contain

/var/log/lwsws/*log {
copytruncate
missingok
notifempty
delaycompress
}

You can find this prepared in /lwsws/etc-logrotate.d-lwsws

Prepare the log directory like this

sudo mkdir /var/log/lwsws
sudo chmod 700 /var/log/lwsws

Debugging lwsws with gdb

Hopefully you won't need to debug lwsws itself, but you may want to debug your plugins. start lwsws like this to have everything running under gdb

sudo gdb -ex "set follow-fork-mode child" -ex "run" --args /usr/local/bin/lwsws

this will give nice backtraces in lwsws itself and in plugins, if they were built with symbols.

Running lwsws under valgrind

You can just run lwsws under valgrind as usual and get valid results. However the results / analysis part of valgrind runs after the plugins have removed themselves, this means valgrind backtraces into plugin code is opaque, without source-level info because the dynamic library is gone.

There's a simple workaround, use LD_PRELOAD=<plugin.so> before running lwsws, this has the loader bring the plugin in before executing lwsws as if it was a direct dependency. That means it's still mapped until the whole process exits after valgtind has done its thing.