 |
libwebsockets
Lightweight C library for HTML5 websockets
|
|
libwebsockets
Lightweight C library for HTML5 websockets
|
When something goes wrong between several machines, eg, the nodes of a DHT and the server they talk to, the logs that explain it are spread across all of them, each in its own journal or file, each on its own clock. Finding out what happened means collecting them by hand, if they still exist, and lining them up.
Log shipping lets a process mirror its lws log lines to a log ingestor, which keeps the lines of many processes on many machines in one sqlite db, in UTC, so they can be looked at together: by a person in a browser, side by side at the same instant, or by an agent over a JSON api.
It's off until someone asks for it. The processes stay connected to the ingestor, but send nothing until a session asks for their lines, from a UTC time that may be a little in the past, for a limited time, and if wanted, with more logging than usual while it's open.
Every lws log line the process logs, at whatever level its log context lets through, from whatever thread, whichever log context it went through and wherever it was emitted to: stderr, syslog and the journal, a file. So it's the same on Linux, Windows and macOS. What isn't an lws log line, like coredumps, OOM kills or the service manager restarting the process, isn't there.
The lines go into a backlog in memory, 4MiB by default, with their UTC time to the us and a sequence number. lws's own local time and level prefix are taken off, anything not printable ASCII becomes ?, and lines are cut at 1024 chars. When it's full, the oldest go. A line stays in the backlog until the ingestor has stored it, or it's pushed out.
When lws's log spew handling stops emitting lines because there are too many, the shipper still keeps them: they may be exactly the ones wanted.
A shipper is identified by
A session is
The ingestor gives each shipper the union of the windows of the open sessions that name it. The shipper sends the lines in its backlog from the window's start, and every line it logs until the window's end, each once. If the window asks for more logging, the process logs more until the window closes, everywhere it logs to, its journal too; then it goes back to what it was.
Raising the level only affects lines logged from then on: lines in the backlog were only kept at the level in force when they were logged. And only levels built into the library can be raised to: a release build of lws doesn't have info or debug logs in it.
A window closes by itself at its end, even if the shipper can't hear from the ingestor, and in any case at most 24h after it was given.
Sessions overlap at no cost: a line is stored once, and a session is just a window of time and the sources it named.
Some fleets want every line all the time, eg, sai's daemons. The ingestor's always pvo names sources, as patterns like a session's src, that are always given a window from the start of their backlog until a while ahead, which moves on every half hour. So they send everything they've kept when they connect, and every line after, as they did before. Sessions still apply to them too, eg, to raise their level for a while.
A line may have a tag: up to 32 of [A-Za-z0-9._/-] saying what it's about, eg, a job. An application gives the shipper a tagger with lws_log_ship_set_tagger(), a function that looks at a line as it's sent and says its tag, if it has one. The ingestor keeps it with the line, and when each tag was first and last seen, so the lines about something can be found by it, and when to look for them. A tag a also finds the lines tagged under it, a/b.
The shipper keeps a websocket connection to the ingestor, subprotocol lws-log-ingest, retrying with backoff if it can't make it, and pinging it every 30s. It must be wss://, since the first thing it sends is the shared secret; ws:// is only allowed to an address on the same machine. The ingestor's certificate is checked like any other client connection on the vhost.
What's said on it, each a ws text message:
Times are UTC us. l is the line's LLL_ bit. A batch is up to about 48KiB of JSON, and only one is outstanding at a time; lines are freed when the ack for them comes. A batch lost with the connection, or not acked within 30s, is sent again, and the ingestor ignores lines it already has.
Every vhost has the shipper's lws-log-ship protocol, and a vhost given pvos for it makes the shipper, connecting from that vhost with its client tls settings. In lwsws, that's in the vhost's ws-protocols, like any protocol's options:
| pvo | default | what it is |
|---|---|---|
| url | the ingestor, wss://, or ws:// to this machine | |
| secret | shared with the ingestor, up to 256 chars | |
| kind | lws | what kind of process this is, eg, lwsws |
| host | the hostname | the name it ships as |
| backlog-kb | 4096 | how much of its lines it keeps, 1 to 1048576 |
The config file holds the secret, so keep it readable only by root. There can only be one shipper in a process, so only one vhost may have these pvos. A process that also runs the ingestor needs none: its lines are kept by the ingestor itself (own-logs), and these pvos there just say so and do nothing. An application not using lwsws can give them in its vhost's info.pvo the same way.
It's destroyed with its vhost, or by lws_log_ship_destroy(). See include/libwebsockets/lws-log-ship.h.
A process that wants its lines from when it started, but only knows where they go later, eg, once its config is read and its network is up, makes the shipper early without a url, and gives it one with lws_log_ship_link() when it can. Until then, its lines wait in the backlog.
A process that only learns its name later calls lws_log_ship_set_host(): it says hello again with the new name, and the ingestor renames the source, with what it already has from it, since it's the same inst.
The lws-log-ingest plugin, on an lwsws vhost:
The shippers connect to any path that isn't behind the interceptor, eg /lws-log-ship; they're authenticated by the secret, not by lws-login.
A process other than lwsws that loads the plugin must link the shared lws library, as the plugin does. Linked to the static one, there are two lws in the process, each with its own logging, and the ingestor only sees the lines logged through its own copy: the process' own lines are silently missing.
| pvo | default | what it is |
|---|---|---|
| database | the sqlite db, made if it isn't there | |
| secret | shared with the shippers, up to 256 chars | |
| Built-in service loop entry | lws-logs | the lws-login grant the api is for |
| retention-days | 14 | lines and sessions older than this go, hourly |
| rate-lines | 12000 | the most lines a shipper may store a minute; past it they're dropped and a line says how many |
| max-session-secs | 3600 | the longest a session may last |
| max-past-secs | 3600 | the furthest in the past a session may start |
| own-logs | 1 | keep this process' own lines too, unless it already has a shipper; they're listed as connected, marked local |
| own-kind | lws-log-ingest | the kind this process' own lines are kept as |
| always | sources always in a window, see "Always" | |
| trust-login-headers | 0 | see "Who can use the api" |
Whatever a shipper sends is checked as if it came from anywhere: the secret only says it's one of ours, not that what it says is sane. A message may be at most 256KiB; a line must have a single real LLL_ bit, be printable ASCII of up to 1024 chars, and be within a day of the ingestor's clock; a shipper whose inst was seen before as a different kind is refused.
In WAL mode, so something else may read it while the ingestor writes:
Each line keeps its shipper's own timestamp, and rx_us when the ingestor stored it. A source's lag_us is the smallest rx_us - t_us of its latest batch: its clock's skew, plus how long its lines waited to go. Read a source's times with that in mind. Lines the ingestor adds itself, eg, to say some were dropped, have no seq.
Under the api mount, answers are JSON. Times in queries are UTC ISO 8601 (2026-10-14T08:01:02Z, optionally with fractional seconds), epoch ms, or -<secs> before now. A refusal of a request that was understood says why, as {"error":"..."}.
GET sessions lists the latest 50, GET sessions/<id> one:
POST sessions starts one, with a JSON body, all of it optional, and Content-Type: application/json:
from is now by default, duration 600s, lvl nothing more than is logged anyway, src everything. The answer is the new session.
POST sessions/<id>/stop, with the body {}, ends one early.
Only JSON bodies are taken: a page elsewhere can't make a logged-in browser send one without asking first, which the ingestor never agrees to.
GET sources says which shippers are connected, by their source id, and until when each is in a window, and the sources heard from in the last day. The ingestor's own process, when it keeps its own lines, is listed first among the connected with "local":true, and an id of null until it has stored some.
A refusal says why as {"error":"..."}.
GET logs:
| arg | what it is |
|---|---|
| session | the session, whose window and src are the defaults |
| from, to | the window; without a session, the last 10 minutes |
| src | comma-separated kinds or hosts, up to 8, * matches anything |
| lvl | the least important level wanted: err, warn, notice, info or debug; notice and below include user lines |
| grep | a substring (not a regex) the line must contain |
| tag | the tag the line must have, or be under |
| limit | the most lines to answer with, 2000 by default, at most 10000 |
| after | the next of the last answer, to get the lines after it |
Lines come in UTC order; i tells lines of the same time apart. l is the line's LLL_ bit: 1 err, 2 warn, 4 notice, 8 info, 16 debug, 1024 user. g is its tag, or null. An answer stops at limit lines or 8MiB, with truncated true. next is after the last line given, to ask for the rest with, or, for a window still open, for what came since.
GET tags/<tag> says when the lines with the tag, or under it, were first and last seen:
For looking at every source's lines at the same instant, a pane each, the viewer asks for what's in a pane a piece at a time. Times are UTC us.
GET sync?session=<id> (or from= and to=, and src=), optionally &at=<us>, says what panes there are:
start_us to end_us is the window, up to now while it's open; from_us to to_us is that padded by 30s. A pane is a kind and host with lines in it; insts says how many instances of it there were, eg, it restarted.
GET pane?pane=<kind>/<host>&at=<us>[&before=<n>][&after=<n>] gives a pane's lines either side of the instant, up to before at or before it and after after it, 150 each by default and at most 500:
lws-login on the mount decides, by the grant named in its service-name, lws-logs above. It stamps the result on the request, and the ingestor answers a user of that grant, or a global admin, and refuses everyone else with 403. It checks the stamped service is its own Built-in service loop entry, so if the mount in front of it is missing or for another grant, it refuses. Sessions record the x-lws-login-sub of who started them.
The stamp is read from inside lwsws, lws_http_get_onward_header(), so the ingestor must be served by the same lwsws as lws-login. If instead it sits behind a front end that proxies to it, eg, on a unix socket, set trust-login-headers to read them from the request: then nothing else may be able to reach the ingestor, since anyone who can could send their own.
Agents use an account given the grant, and log in like a browser:
see its --help.
The plugin's assets, mounted as above, are a page for starting, stopping and choosing sessions, seeing which shippers are connected, and looking at a session's lines: every source side by side at one instant, or merged in UTC order. #session=<id> on its url opens a session, so other pages can link to one.
On each DHT node's lwsws, add lws-log-ship to the ws-protocols of the vhost with the DHT, as above, and on the server, the ingestor. Nothing is sent until a session is started, eg, to see the last two minutes and the next ten of every node, with the nodes logging at info meanwhile:
1.18.0