Log shipping

Data Structures
struct	lws_log_ship_line
struct	lws_log_ship_info

Macros
#define	LWS_LOG_SHIP_WS_PROTOCOL   "lws-log-ingest"
#define	LWS_LOG_SHIP_LINE_MAX   1024

Typedefs
typedef struct lws_log_ship_line	lws_log_ship_line_t
typedef int(*	lws_log_ship_tagger_t) (const char *line, size_t len, char *tag, size_t tag_len)
typedef void(*	lws_log_ship_store_cb_t) (void *opaque, const char *kind, const char *host, const char *inst, const lws_log_ship_line_t *lines, size_t count)

Functions
LWS_VISIBLE LWS_EXTERN struct lws_log_ship *	lws_log_ship_create (const struct lws_log_ship_info *info)
LWS_VISIBLE LWS_EXTERN int	lws_log_ship_link (struct lws_log_ship *ls, const char *url, const char *secret)
LWS_VISIBLE LWS_EXTERN int	lws_log_ship_set_host (struct lws_log_ship *ls, const char *host)
LWS_VISIBLE LWS_EXTERN void	lws_log_ship_set_tagger (lws_log_ship_tagger_t tagger)
LWS_VISIBLE LWS_EXTERN void	lws_log_ship_destroy (struct lws_log_ship **pls)
LWS_VISIBLE LWS_EXTERN void	lws_log_ship_window (struct lws_log_ship *ls, uint64_t from_us, uint64_t until_us, int lvl)
LWS_VISIBLE LWS_EXTERN struct lws_log_ship *	lws_log_ship_get (void)
LWS_VISIBLE LWS_EXTERN const char *	lws_log_ship_inst (struct lws_log_ship *ls)
LWS_VISIBLE LWS_EXTERN const char *	lws_log_ship_host (struct lws_log_ship *ls)
LWS_VISIBLE LWS_EXTERN uint64_t	lws_log_ship_utc_us (void)

Detailed Description

Log shipping

Mirrors the lws log lines this process emits to a log ingestor, eg, the lws-log-ingest plugin, which keeps the lines of many processes on many machines in one place, in UTC, so they can be looked at together. See READMEs/README.log-ship.md.

Every line lws emits, whichever log context it went through and wherever it went (stderr, syslog / the journal, a file), is also kept in a bounded backlog in memory with its UTC time. Nothing is sent by default: the process keeps a ws connection open to the ingestor, authenticated by a shared secret, and the ingestor tells it when to send. It names a window, from a UTC time that may be a little in the past, which the backlog provides, until a UTC time a limited time ahead; while the window is open it may also raise the process' log level, and when it closes the level goes back to what it was.

There is one log shipper per process, since the lines it sees are the whole process'.

Besides lws_log_ship_create(), a vhost makes one when it's given pvos for the "lws-log-ship" protocol, which every vhost has: "url" and "secret", and optionally "kind" ("lws" by default), "host" and "backlog-kb". That's how lwsws makes one, from a vhost's "ws-protocols" config.

---

Data Structure Documentation

lws_log_ship_line

struct lws_log_ship_line

lws_log_ship_line_t - a log line, as given to a local store

Definition at line 58 of file lws-log-ship.h.

Collaboration diagram for lws_log_ship_line:

Data Fields
uint64_t	seq	per-shipper sequence, from 1
uint64_t	t_us	UTC us when it was logged
const char *	text	printable ASCII, NUL terminated
const char *	tag	NULL, or its tag, see lws_log_ship_set_tagger()
size_t	len	strlen(text)
int	level	its single LLL_ bit

lws_log_ship_info

struct lws_log_ship_info

struct lws_log_ship_info - how to ship our logs

Definition at line 105 of file lws-log-ship.h.

Collaboration diagram for lws_log_ship_info:

Data Fields
struct lws_vhost *	vhost	the vhost the connection to the ingestor is made on, its client tls settings apply. The shipper is destroyed with the vhost, if lws_log_ship_destroy() wasn't called before.
const char *	url	the ingestor, "wss://host[:port]/path"; ws:// is only allowed to a loopback address. NULL with .store set for a local store, or NULL with no .store to only keep lines until lws_log_ship_link() says where they go, eg, when that's only known later.
const char *	secret	shared with the ingestor, it's sent inside the tls tunnel
const char *	kind	what kind of process we are, eg "lwsws", up to 32 of [A-Za-z0-9._-]
const char *	host	our name, up to 64 of [A-Za-z0-9._-], NULL means the hostname
size_t	backlog_max	bytes of lines kept for windows that start in the past, and while they're waiting to be sent; 0 means 4MiB. Past it, the oldest go
lws_log_ship_store_cb_t	store	NULL, or a local store, used instead of a connection
void *	opaque	passed to .store
int	client_ssl_flags	extra LCCSCF_ flags for the connection, eg, to allow a selfsigned ingestor in a test
char	no_sysinfo	the first window starts with lines describing the system, see lws_sysinfo_create(), unless this is set, eg, because the application logs them itself

Macro Definition Documentation

LWS_LOG_SHIP_WS_PROTOCOL

#define LWS_LOG_SHIP_WS_PROTOCOL   "lws-log-ingest"

#include <lws-log-ship.h>

Definition at line 253 of file lws-log-ship.h.

LWS_LOG_SHIP_LINE_MAX

#define LWS_LOG_SHIP_LINE_MAX   1024

#include <lws-log-ship.h>

Definition at line 254 of file lws-log-ship.h.

Typedef Documentation

lws_log_ship_line_t

typedef struct lws_log_ship_line lws_log_ship_line_t

#include <lws-log-ship.h>

lws_log_ship_line_t - a log line, as given to a local store

lws_log_ship_tagger_t

typedef int(* lws_log_ship_tagger_t) (const char *line, size_t len, char *tag, size_t tag_len)

#include <lws-log-ship.h>

lws_log_ship_tagger_t - find a line's tag

Parameters

line	the line, printable ASCII
len	its length
tag	where to put the tag, NUL terminated
tag_len	the room at tag, 33

Returns 0 if the line has a tag, else nonzero. A tag is 1 to 32 of [A-Za-z0-9._/-], anything else is ignored. It's called with the backlog locked, so it must be quick, and mustn't log.

Definition at line 80 of file lws-log-ship.h.

lws_log_ship_store_cb_t

typedef void(* lws_log_ship_store_cb_t) (void *opaque, const char *kind, const char *host, const char *inst, const lws_log_ship_line_t *lines, size_t count)

#include <lws-log-ship.h>

lws_log_ship_store_cb_t - a local store takes some lines

Parameters

opaque	the info's .opaque
kind	the shipper's kind, eg "lwsws"
host	the shipper's host name
inst	the shipper's instance, 16 lowercase hex
lines	the lines, in seq order
count	how many lines

For a shipper in the ingestor's own process, which stores its lines without a connection. It's called from the event loop, and must not hold on to the lines after it returns.

Definition at line 97 of file lws-log-ship.h.

Function Documentation

lws_log_ship_create()

LWS_VISIBLE LWS_EXTERN struct lws_log_ship * lws_log_ship_create

(

const struct lws_log_ship_info *

info

)

#include <lws-log-ship.h>

lws_log_ship_create() - start keeping our log lines, and connect to the ingestor

Parameters

info	how to ship our logs, its strings are copied

Returns the shipper, or NULL if it couldn't be made, or there already is one in this process.

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_link()

LWS_VISIBLE LWS_EXTERN int lws_log_ship_link	(	struct lws_log_ship *	ls,
		const char *	url,
		const char *	secret )

#include <lws-log-ship.h>

lws_log_ship_link() - say where the lines of a shipper made without a url go

Parameters

ls	the shipper
url	the ingestor, as for lws_log_ship_info .url
secret	shared with the ingestor

Returns 0 if it's connecting, or nonzero if it already has somewhere to go, or the url or secret can't be used.

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_set_host()

LWS_VISIBLE LWS_EXTERN int lws_log_ship_set_host	(	struct lws_log_ship *	ls,
		const char *	host )

#include <lws-log-ship.h>

lws_log_ship_set_host() - ship as a different host name from now on

Parameters

ls	the shipper
host	the new name, up to 64 of [A-Za-z0-9._-]

Eg, a process that only learns its name some time after it started. The ingestor renames the source, with what it already has from it.

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_set_tagger()

LWS_VISIBLE LWS_EXTERN void lws_log_ship_set_tagger

(

lws_log_ship_tagger_t

tagger

)

#include <lws-log-ship.h>

lws_log_ship_set_tagger() - tag lines as they're sent

Parameters

tagger

NULL, or the tagger

An application can tag lines with something they're about, eg, a job, by what's in them, so the ingestor can find them by it. It's processwide, and can be set before the shipper is made.

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_destroy()

LWS_VISIBLE LWS_EXTERN void lws_log_ship_destroy

(

struct lws_log_ship **

pls

)

#include <lws-log-ship.h>

lws_log_ship_destroy() - stop keeping and shipping our log lines

Parameters

pls	the shipper, set to NULL

Lines not yet stored by the ingestor are lost. A raised log level goes back to what it was.

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_window()

LWS_VISIBLE LWS_EXTERN void lws_log_ship_window	(	struct lws_log_ship *	ls,
		uint64_t	from_us,
		uint64_t	until_us,
		int	lvl )

#include <lws-log-ship.h>

lws_log_ship_window() - open, change or close the window of lines to send

Parameters

ls	the shipper
from_us	UTC us of the earliest line wanted
until_us	UTC us when the window closes, 0 closes it now
lvl	LLL_ bits to log as well as the usual ones while it's open

The ingestor does this for a connected shipper; this is for a local store, or an application that decides the window itself. The lines in the backlog logged from from_us on, and every line logged until until_us, are sent, each once. A new window replaces the last one. Whoever asks, a window closes at most 24h from when it was given.

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_get()

LWS_VISIBLE LWS_EXTERN struct lws_log_ship * lws_log_ship_get

(

void

)

#include <lws-log-ship.h>

lws_log_ship_get() - the process' shipper, if it has one

Returns the shipper, or NULL. There can only be one, so something that would make one, eg, from config, can see first that the process' lines already go somewhere.

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_inst()

LWS_VISIBLE LWS_EXTERN const char * lws_log_ship_inst

(

struct lws_log_ship *

ls

)

#include <lws-log-ship.h>

lws_log_ship_inst() - our instance name, 16 lowercase hex

Parameters

ls	the shipper

It's chosen at random when the shipper is created, so lines from a restarted process, or a reused host name, can be told apart.

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_host()

LWS_VISIBLE LWS_EXTERN const char * lws_log_ship_host

(

struct lws_log_ship *

ls

)

#include <lws-log-ship.h>

lws_log_ship_host() - the host name we ship as

Parameters

ls	the shipper

References LWS_EXTERN, and LWS_VISIBLE.

lws_log_ship_utc_us()

LWS_VISIBLE LWS_EXTERN uint64_t lws_log_ship_utc_us

(

void

)

#include <lws-log-ship.h>

lws_log_ship_utc_us() - the UTC time now, in us, as the shipper takes it