[Libwebsockets] Help wanted: V4L2 / mp4 / h.264

Andy Green andy at warmcat.com
Fri May 14 15:18:20 CEST 2021

On 5/14/21 1:25 PM, Rémi COHEN-SCALI ... wrote:
> Hi guys
> Wanted to know if you found someone for helping on H.264/mp4 container 
> support for your lib.

No, I am very happy to see your email.

> I worked 8 years for Nagravision (CAS vendor: this is the Conditional 
> Access System part of the MPEG spec, that is not specified by MPEG ... 
> lol), mostly for the embedded part and I had to cope with many mpeg 
> specs including iso container and video/audio codecs (h264, hevc, aac). 
> I developed or made specs for some very different things as real time 
> video watermaking, secured stream deciphering in Android or chrome with 
> nagra drm, smart cards security ... I have a pretty good knowledge of 
> ISO BMFF container (ISO/IEC 14496-3), MP4 file format (ISO/IEC 
> 14496-14), AVC (h264: ISO/IEC 14496-15), HEIF (jpg2000) and HEVC (h265).


The basic v4l2 part and MJPG live streaming was not that difficult, so I 
thought wiring up libav* to do the transcoding and mp4 container would 
just be a bit harder.  It's not that hard to do the transcoding for 
video only, but the standards situation vs empirical "what do the 
browsers accept" situation is not good.  Which is where it's stuck, 
working on Firefox but not Chrome for live video; both Firefox and 
Chrome can play the same video spooled to disk.

What I set out to do is get the stuff working and then refactor it to 
allow various purposes, eg, v4l2 -> MJPG -> h.264 -> storage; v4l2 -> 
MJPG -> h.264 -> live stream... the use cases I have for it are 
integrate it into CI, so you can see what happened during the tests 
sync'd with the logs, which is a mashup of the storage and "live" 
playback from that; for some devices it's also interesting to use a 
HDMI->USB + keyboard gadget to get remote access to them securely in 
realtime in the browser, even for hard situations like need to operate 
BIOS, so you would be looking at h.264 live video and typing things in 
the browser that appear as USB keyboard input.

The same pieces can also be used for n clients connecting and viewing a 
live stream.

It also became clear, while libav* is valuable in that it abstracts away 
codec selection etc but if you are willing to just specify mp4 + 
h.264/5, it is in fact optional, since I had to get my hands so far into 
mp4 there is now a set of lightweight pieces for the whole moov / moof 
generation, that could be wired up to, eg, a baremetal h.264 encoder 
without libav*.  Some mp4 info has to be redone or inserted according to 
if a client joins the stream anyway.

> However I'm very interested in your project. It is really a very nice 
> work. Then I'd be very happy to help, if I can. I'll have a look at your 
> docs and will try to understand your proposal around isobmff. I someone 
> is already looking at this perhaps I can help is some way (and ask him 
> questions). If not, if you don't mind, I'll send my question to this list.
> Then I will certainly be back here, seeking for answers :)

Yes feel free to ask here.

I updated the _v4l2 branch with the latest stuff from last weekend.  The 
code is in here


The objects are defined in private.h, notice there is a #define at the 
minute called OWN_MOOF that is set, this bypasses libav mux stream 
generation (which does not succeed to stream to Chrome either) and 
instead takes the h.264 encoder packets and copies and strips the annex 
b / TS stuff into buffers in struct src_inst, and puts them back 
according to client state rather than source video state.  If it's not 
set, it uses libav streams to do the muxing.

annex-b.c in there parses that and produces the avcC format blob from 
the SPS and PPS.

boxes.c became a complete box generation implementation which maintains 
a stack of parents and handles their length automatically.  Because it 
became easier to do that than try to continue fiddling them by hand.

transcode.c takes the MJPEG as it comes and passes it through libav* as 
it stands, which passes it through libx264 and produces an AVPacket 
combining annex-b with h.264

v4l2.c handles setting up the stream as MJPEG, it wants to prefer native 
h.264 if available but we are not at that point yet.\

minimal-ws-server.c has an important define DUMP_TO_FILE, if set, it 
will take a copy of the first video client data to a file /tmp/str.mp4. 
  You can use this to confirm the output is OK in mplayer / vlc etc as 
well as by file:/// that it can be played in the browsers.

To test just install the libav* dev packages and build lws with 
LWS_WITH_MINIMAL_EXAMPLES=1, you will get a 
./build/bin/lws-minimal-ws-server-v4l2 that you should run from the cwd 

Then visit in your browser, it defaults to trying to 
stream /dev/video0.


> Thanks guys... and keep up good work !
> Regards
> Rémi
> _______________________________________________
> Libwebsockets mailing list
> Libwebsockets at ml.libwebsockets.org
> https://libwebsockets.org/mailman/listinfo/libwebsockets

More information about the Libwebsockets mailing list