[Libwebsockets] Lightweight JSON parser in C

"Andy Green (林安廸)" andy at warmcat.com
Sat Feb 23 02:14:15 CET 2013


On 23/02/13 00:36, the mail apparently from Jack Mitchell included:
> On 22/02/13 16:11, "Andy Green (林安廸)" wrote:
>> On 22/02/13 23:53, the mail apparently from Jack Mitchell included:
>>> On 22/02/13 15:28, "Andy Green (林安廸)" wrote:
>>>> Hi -
>>>>
>>>> Is there any interest in a very lightweight but 100% correct JSON
>>>> parser?
>>>>
>>>> I don't mean like you have in a browser where it goes through and
>>>> spawns objects for everything in the JSON. The idea is it is basically
>>>> a stateful stream reader, which can do something when the object path
>>>> it is on top of matches what you're interested in.
>>>>
>>>> It does not recurse, nor malloc anything. You call it with cursor
>>>> structs which start off zero'd and hold all parsing state. The cursor
>>>> structs have no pointers and may be copied. You can call the multiple
>>>> times with the same cursor as new JSON pieces come in and it will
>>>> parse on from the previous state; it's a bytewise parser so there are
>>>> no fragmentation issues. It means you don't need to hold the JSON all
>>>> at once anywhere but can still parse the whole JSON.
>>>>
>>>> So the style would be parse until the node of interest is held in the
>>>> cursor struct, then you can use a copy of that as a starting point for
>>>> rummaging around.
>>>>
>>>> I have the parser working - it is very lightweight, the cursor struct
>>>> is the only storage and it's < 256 bytes - but I am a bit stumped
>>>> about how to most flexibly expose the wanted results. I know what I
>>>> want it for and can hack that up easily, but I am wondering if there
>>>> are other real-world uses for very cheap JSON parsing in C... if so,
>>>> what does the JSON look like and what are the operations needed on it?
>>>>
>>>> -Andy
>>>> _______________________________________________
>>>> Libwebsockets mailing list
>>>> Libwebsockets at ml.libwebsockets.org
>>>> http://ml.libwebsockets.org/mailman/listinfo/libwebsockets
>>>
>>> Hi Andy,
>>>
>>> I have JSON parsing in my application and use the Jansson JSON Library
>>> to parse and manage it. A lot of what you said went over my head so I'm
>>> not sure if it would be applicable in my case, as I do a lot of parsing
>>> of the JSON and I also reply to the recieve method in JSON which I
>>> create on the fly.
>>>
>>> If you want to poke into my use case then a few simple fairly specific
>>> questions would probably be enough to coax some useful information out
>>> of me ;)
>>
>> Jansson seems to take the approach to sit down and allocate a parsed
>> model of the JSON, which you can then query, like a browser does it.
>> It's fine but it does not sit well with very resource-constrained
>> devices.  My target is a Cortex-M0 with 32KByte flash and 2KBytes free
>> memory.
>>
>> This is different, it's a stream parser that does not even need the
>> JSON in one place at one time, and it does not "make notes as it
>> goes", it just goes through the JSON like a text search until it sees
>> it has arrived at the bit you are interested in.  (The structure of
>> where it is, the parsing state, is held in a struct.)  Then, you will
>> be able to get parsed elements from it, skip to the next bit you are
>> interested in etc.
>>
>> What I was asking is
>>
>>  - what kind of queries are you doing at the moment in Jansson?
>
> I have a custom RPC style implementation in JSON, so I currently take
> received messages with a method and parameters which I then parse, act
> upon and build a new JSON string to write out to all clients with
> updated information.
>
> I quite often pick parameters out in an odd order and often make
> extensive use of arrays and objects.
>
>>
>>  - Find a section like "thing.mylist", and then get strings from
>> "thing.mylist[n]"?
>
> Yes, my main use is like this, for example this is a verbose snippet of
> my JSON in and out from the server point of view:
>
> [946949789:3863] INFO: insert_wsi_socket_into_fds: wsi=0x1d73c8,
> sock=22, fds pos=1
> [946949789:4391] INFO: Allocating RX buffer 1046
> JSON IN: { "method":"setCard", "card":0 }
> JSON IN: {"method":"0","parameters":{"callbackFunction":"test"}}
> JSON OUT:
> {"success":0,"method":["callback"],"calledParams":{"callbackFunction":"test"}}

Canned JSON out is pretty trivial I think, unless I just didn't find 
anything too complex yet and am left without appreciating the 
complexity.  Here I am just doing it sending a mixture of canned strings 
and live data sent one by one.

The examples of JSON you want to parse at the server boils down to

> JSON IN: {"method":"1","parameters":{"callbackFunction":"test"}}
> JSON IN: {"method":"1","parameters":{"callbackFunction":"test"}}
> JSON IN:
> {"method":"3","parameters":{"routeID":2,"routeState":1,"callbackFunction":"finishSetRoute"}}
> JSON IN:
> {"method":"3","parameters":{"routeID":1,"routeState":1,"callbackFunction":"finishSetRoute"}}
ferent?  Are sub-objects in the results you want? How
>> do you deal with that at the moment?
>
> As you can see above, I do use sub-objects. I'm not so sure my use case
> is something to be fulfilled by a zero-copy speedy parser :)

I think it'll do all that in a good way.  To leverage the stream 
qualities it's preferable if you're using data from the JSON in the 
order it's coming from the JSON... although that sounds like a 
restriction both practically

	if (json.mylist) {
		n = 0;
		while (json.mylist[n]) {
			...
			n++;
		}
	}

...or...

	if (json.important_flag_toplevel) {

		/* do inner work based on flag */

	}

and by habit people tend to read the JSON top-down and do things in that 
order.

I'll post some sample code using a results api when I have it, thanks 
for the extra info in the meanwhile.

-Andy




More information about the Libwebsockets mailing list