libwebsockets
Lightweight C library for HTML5 websockets
RFC8949 CBOR Stream Parsing and Writing

||| |—|—|—| |cmake| LWS_WITH_CBOR, LWS_WITH_CBOR_FLOAT| |Header| ./include/libwebsockets/lws-lecp.h| |api-test| ./minimal-examples/api-tests/api-test-lecp/| |test app| ./test-apps/test-lecp.c -> libwebsockets-test-lecp|

LECP is the RFC8949 CBOR stream parsing counterpart to LEJP for JSON.

Features

  • Completely immune to input fragmentation, give it any size blocks of CBOR as they become available; 1 byte, or 100K at a time give identical parsing results
  • Input chunks discarded as they are parsed, whole CBOR never needed in memory
  • Nonrecursive, fixed stack usage of a few dozen bytes
  • No heap allocations at all, just requires ~500 byte context usually on caller stack
  • Creates callbacks to a user-provided handler as members are parsed out
  • No payload size limit, supports huge / endless strings or blobs bigger than system memory
  • Collates utf-8 text and blob payloads into a 250-byte chunk buffer for ease of access
  • Write apis don't use any heap allocations or recursion either
  • Write apis use an explicit context with its own lifecycle, and printf style vaargs including sized blobs, C strings, double, int, unsigned long etc
  • Completely immune to output fragmentation, supports huge strings and blobs into small buffers, api returns to indicates unfinished if it needs to be called again to continue; 1 byte or 100K output buffer give same results
  • Write apis completely fill available buffer and if unfinished, continues into same or different buffer when called again with same args; no requirement for subsequent calls to be done sequentially or even from same function

Type limits

CBOR allows negative integers of up to 64 bits, these do not fit into a uint64_t. LECP has a union for numbers that includes the types uint64_t and int64_t, but it does not separately handle negative integers. Only -2^63.. 2^64 -1 can be handled by the C types, the oversize negative numbers wrap and should be avoided.

Floating point support

Floats are handled using the IEEE memory format, it means they can be parsed from the CBOR without needing any floating point support in the build. If floating point is available, you can also enable LWS_WITH_CBOR_FLOAT and a float and double types are available in the number item union. Otherwise these are handled as ctx->item.u.u32 and ctx->item.u.u64 union members.

Half-float (16-bit) is defined in CBOR and always handled as a uint16_t number union member ctx->item.u.hf.

Callback reasons

The user callback does not have to handle any callbacks, it only needs to process the data for the ones it is interested in.

Callback reason CBOR structure Associated data
LECPCB_CONSTRUCTED Created the parse context
LECPCB_DESTRUCTED Destroyed the parse context
LECPCB_COMPLETE The parsing completed OK
LECPCB_FAILED The parsing failed
LECPCB_VAL_TRUE boolean true
LECPCB_VAL_FALSE boolean false
LECPCB_VAL_NULL explicit NULL
LECPCB_VAL_NUM_INT signed integer ctx->item.u.i64
LECPCB_VAL_STR_START A UTF-8 string is starting
LECPCB_VAL_STR_CHUNK The next string chunk ctx->npos bytes in ctx->buf
LECPCB_VAL_STR_END The last string chunk ctx->npos bytes in ctx->buf
LECPCB_ARRAY_START An array is starting
LECPCB_ARRAY_END An array has ended
LECPCB_OBJECT_START A CBOR map is starting
LECPCB_OBJECT_END A CBOR map has ended
LECPCB_TAG_START The following data has a tag index ctx->item.u.u64
LECPCB_TAG_END The end of the data referenced by the last tag
LECPCB_VAL_NUM_UINT Unsigned integer ctx->item.u.u64
LECPCB_VAL_UNDEFINED CBOR undefined
LECPCB_VAL_FLOAT16 half-float available as host-endian uint16_t ctx->item.u.hf
LECPCB_VAL_FLOAT32 float (uint32_t if no float support) available ctx->item.u.f
LECPCB_VAL_FLOAT64 double (uint64_t if no float support) available ctx->item.u.d
LECPCB_VAL_SIMPLE CBOR simple ctx->item.u.u64
LECPCB_VAL_BLOB_START A binary blob is starting
LECPCB_VAL_BLOB_CHUNK The next blob chunk ctx->npos bytes in ctx->buf
LECPCB_VAL_BLOB_END The last blob chunk ctx->npos bytes in ctx->buf

|LECPCB_ARRAY_ITEM_START|A logical item in an array is starting| |LCEPDB_ARRAY_ITEM_END|A logical item in an array has completed|

CBOR indeterminite lengths

Indeterminite lengths are supported, but are concealed in the parser as far as possible, the CBOR lengths or its indeterminacy are not exposed in the callback interface at all, just chunks of data that may be the start, the middle, or the end.

Handling CBOR UTF-8 strings and blobs

When a string or blob is parsed, an advisory callback of LECPCB_VAL_STR_START or LECPCB_VAL_BLOB_START occurs first. The _STR_ callbacks indicate the content is a CBOR UTF-8 string, _BLOB_ indicates it is binary data.

Strings or blobs may have indeterminite length, but if so, they are composed of logical chunks which must have known lengths. When the _START callback occurs, the logical length either of the whole string, or of the sub-chunk if indeterminite length, can be found in ctx->item.u.u64.

Payload is collated into ctx->buf[], the valid length is in ctx->npos.

For short strings or blobs where the length is known, the whole payload is delivered in a single LECPCB_VAL_STR_END or LECPCB_VAL_BLOB_END callback.

For payloads larger than the size of ctx->buf[], LECPCB_VAL_STR_CHUNK or LECPCB_VAL_BLOB_CHUNK callbacks occur delivering each sequential bufferload. If the CBOR indicates the total length, the last chunk is delievered in a LECPCB_VAL_STR_END or LECPCB_VAL_BLOB_END.

If the CBOR indicates the string end after the chunk, a zero-length ..._END callback is provided.

Handling CBOR tags

CBOR tags are exposed as LECPCB_TAG_START and LECPCB_TAG_END pairs, at the _START callback the tag index is available in ctx->item.u.u64.

CBOR maps

You can check if you are on the "key" part of a map "key:value" pair using the helper api lecp_parse_map_is_key(ctx).

Parsing paths

LECP maintains a "parsing path" in ctx->path that represents the context of the callback events. As a convenience, at LECP context creation time, you can pass in an array of path strings you want to match on, and have any match checkable in the callback using ctx->path_match, it's 0 if no active match, or the match index from your path array starting from 1 for the first entry.

CBOR element Representation in path
CBOR Array []
CBOR Map .
CBOR Map entry key string keystring

Accessing raw CBOR subtrees

Some CBOR usages like COSE require access to selected raw CBOR from the input stream. lecp_parse_report_raw(ctx, on) lets you turn on and off buffering of raw CBOR and reporting it in the parse callback with LECPCB_LITERAL_CBOR callbacks. The callbacks mean the temp buffer ctx->cbor[] has ctx->cbor_pos bytes of raw CBOR available in it. Callbacks are triggered when the buffer fills, or reporting is turned off and the buffer has something in it.

By turning the reporting on and off according to the outer CBOR parsing state, it's possible to get exactly the raw CBOR subtree that's needed.

Capturing and reporting the raw CBOR does not change that the same CBOR is being passed to the parser as usual as well.

Comparison with LEJP (JSON parser)

LECP is based on the same principles as LEJP and shares most of the callbacks. The major differences:

  • LEJP value callbacks all appear in ctx->buf[], ie, floating-point is provided to the callback in ascii form like "1.0". CBOR provides a more strict typing system, and the different type values are provided either in ctx->buf[] for blobs or utf-8 text strtings, or the item.u union for converted types, with additional callback reasons specific to each type.
  • CBOR "maps" use _OBJECT_START and _END parsing callbacks around the key / value pairs. LEJP has a special callback type PAIR_NAME for the key string / integer, but in LECP these are provided as generic callbacks dependent on type, ie, generic string callbacks or integer ones, and the value part is represented according to whatever comes.

Writing CBOR

CBOR is written into a lws_lec_pctx_t object that has been initialized to point to an output buffer of a specified size, using printf type formatting.

Output is paused if the buffer fills, and the write api may be called again later with the same context object, to resume emitting to the same or different buffer.

This allows bufferloads of encoded CBOR to be produced on demand, it's designed to fit usage in WRITEABLE callbacks and Secure Streams tx() callbacks where the buffer size for one packet is already fixed.

CBOR array and map lengths are deduced from the format string, as is whether to use indeterminite length formatting or not. For indeterminite text or binary strings, a container of < >

Format Arg(s) Meaning
123 unsigned literal number
-123 signed literal number
u unsigned int number
lu unsigned long int number
llu unsigned long long int number
d signed int number
ld signed long int number
lld signed long long int number
f double floating point number
123(...) literal tag and scope
t(...) unsigned int tag and scope
lt(...) unsigned long int tag and scope
llt(...) unsigned long long int tag and scope
[...] Array (fixed len if ] in same format string)
{...} Map (fixed len if } in same format string)
<t...> Container for indeterminite text string frags
<b...> Container for indeterminite binary string frags
‘'string’\ilinebr </td> <td class="markdownTableBodyNone"> Literal text of known length \ilinebr </td> </tr> <tr class="markdownTableRowOdd"> <td class="markdownTableBodyNone">s\ilinebr </td> <td class="markdownTableBodyNone">const char *\ilinebr </td> <td class="markdownTableBodyNone"> NUL-terminated string \ilinebr </td> </tr> <tr class="markdownTableRowEven"> <td class="markdownTableBodyNone">%.*s\ilinebr </td> <td class="markdownTableBodyNone">int,const char *\ilinebr </td> <td class="markdownTableBodyNone"> length-specified string \ilinebr </td> </tr> <tr class="markdownTableRowOdd"> <td class="markdownTableBodyNone">%.*b\ilinebr </td> <td class="markdownTableBodyNone">int,const uint8_t *\ilinebr </td> <td class="markdownTableBodyNone"> length-specified binary \ilinebr </td> </tr> <tr class="markdownTableRowEven"> <td class="markdownTableBodyNone" colspan="2">:\ilinebr </td> <td class="markdownTableBodyNone"> separator between Map items (a:b) \ilinebr </td> </tr> <tr class="markdownTableRowOdd"> <td class="markdownTableBodyNone" colspan="2">,` separator between Map pairs or array items

Backslash is used as an escape in ‘’...'literal strings, so'\'represents a string consisting of a single backslash, and'\''` a string consisting of a single single-quote.

For integers, various natural C types are available, but in all cases, the number is represented in CBOR using the smallest valid way based on its value, the long or long-long modifiers just apply to the expected C type in the args.

For floats, the C argument is always expected to be a double type following C type promotion, but again it is represented in CBOR using the smallest valid way based on value, half-floats are used for NaN / Infinity and where possible for values like 0.0 and -1.0.

Examples

Literal ints

uint8_t buf[128];
lws_lec_pctx_t cbw;
lws_lec_init(&cbw, buf, sizeof(buf));
lws_lec_printf(ctx, "-1");

||| |—|—| |Return| LWS_LECPCTX_RET_FINISHED| |ctx->used|1| |buf[]|20|

Dynamic ints

uint8_t buf[128];
lws_lec_pctx_t cbw;
int n = -1; /* could be long */
lws_lec_init(&cbw, buf, sizeof(buf));
lws_lec_printf(ctx, "%d", n); /* use %ld for long */

||| |—|—| |Return| LWS_LECPCTX_RET_FINISHED| |ctx->used|1| |buf[]|20|

Maps, arrays and dynamic ints

...
int args[3] = { 1, 2, 3 };
lws_lec_printf(ctx, "{'a':%d,'b':[%d,%d]}", args[0], args[1], args[2]);

||| |—|—| |Return| LWS_LECPCTX_RET_FINISHED| |ctx->used|9| |buf[]|A2 61 61 01 61 62 82 02 03|

String longer than the buffer

Using s and the same string as an arg gives same results

uint8_t buf[16];
lws_lec_pctx_t cbw;
lws_lec_init(&cbw, buf, sizeof(buf));
lws_lec_printf(ctx, "'A literal string > one buf'");
/* not required to be in same function context or same buf,
* but the string must remain the same */
lws_lec_setbuf(&cbw, buf, sizeof(buf));
lws_lec_printf(ctx, "'A literal string > one buf'");

First call

||| |—|—| |Return| LWS_LECPCTX_RET_AGAIN| |ctx->used|16| |buf[]|78 1A 41 20 6C 69 74 65 72 61 6C 20 73 74 72 69|

Second call

||| |—|—| |Return| LWS_LECPCTX_RET_FINISHED| |ctx->used|12| |buf[]|6E 67 20 3E 20 6F 6E 65 20 62 75 66|

Binary blob longer than the buffer

uint8_t buf[16], blob[] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18 };
lws_lec_pctx_t cbw;
lws_lec_init(&cbw, buf, sizeof(buf));
lws_lec_printf(ctx, "%.*b", (int)sizeof(blob), blob);
/* not required to be in same function context or same buf,
* but the length and blob must remain the same */
lws_lec_setbuf(&cbw, buf, sizeof(buf));
lws_lec_printf(ctx, "%.*b", (int)sizeof(blob), blob);

First call

||| |—|—| |Return| LWS_LECPCTX_RET_AGAIN| |ctx->used|16| |buf[]|52 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F|

Second call

||| |—|—| |Return| LWS_LECPCTX_RET_FINISHED| |ctx->used|3| |buf[]|10 11 12|