[Libwebsockets] Help wanted: V4L2 / mp4 / h.264

Rémi COHEN-SCALI ... remi at cohenscali.net
Fri May 14 15:44:22 CEST 2021

Le 14/05/2021 à 15:18, Andy Green a écrit :
> On 5/14/21 1:25 PM, Rémi COHEN-SCALI ... wrote:
>> Hi guys
>> Wanted to know if you found someone for helping on H.264/mp4 container
>> support for your lib.
> No, I am very happy to see your email.
Great! Happy I can help ;-)
>> I worked 8 years for Nagravision (CAS vendor: this is the Conditional
>> Access System part of the MPEG spec, that is not specified by MPEG ...
>> lol), mostly for the embedded part and I had to cope with many mpeg
>> specs including iso container and video/audio codecs (h264, hevc, aac).
>> I developed or made specs for some very different things as real time
>> video watermaking, secured stream deciphering in Android or chrome with
>> nagra drm, smart cards security ... I have a pretty good knowledge of
>> ISO BMFF container (ISO/IEC 14496-3), MP4 file format (ISO/IEC
>> 14496-14), AVC (h264: ISO/IEC 14496-15), HEIF (jpg2000) and HEVC (h265).
> Great.
> The basic v4l2 part and MJPG live streaming was not that difficult, so I
> thought wiring up libav* to do the transcoding and mp4 container would
> just be a bit harder.  It's not that hard to do the transcoding for
> video only, but the standards situation vs empirical "what do the
> browsers accept" situation is not good.  Which is where it's stuck,
> working on Firefox but not Chrome for live video; both Firefox and
> Chrome can play the same video spooled to disk.
> What I set out to do is get the stuff working and then refactor it to
> allow various purposes, eg, v4l2 -> MJPG -> h.264 -> storage; v4l2 ->
> MJPG -> h.264 -> live stream... the use cases I have for it are
> integrate it into CI, so you can see what happened during the tests
> sync'd with the logs, which is a mashup of the storage and "live"
> playback from that; for some devices it's also interesting to use a
> HDMI->USB + keyboard gadget to get remote access to them securely in
> realtime in the browser, even for hard situations like need to operate
> BIOS, so you would be looking at h.264 live video and typing things in
> the browser that appear as USB keyboard input.
> The same pieces can also be used for n clients connecting and viewing a
> live stream.
> It also became clear, while libav* is valuable in that it abstracts away
> codec selection etc but if you are willing to just specify mp4 +
> h.264/5, it is in fact optional, since I had to get my hands so far into
> mp4 there is now a set of lightweight pieces for the whole moov / moof
> generation, that could be wired up to, eg, a baremetal h.264 encoder
> without libav*.  Some mp4 info has to be redone or inserted according to
> if a client joins the stream anyway.
>> However I'm very interested in your project. It is really a very nice
>> work. Then I'd be very happy to help, if I can. I'll have a look at your
>> docs and will try to understand your proposal around isobmff. I someone
>> is already looking at this perhaps I can help is some way (and ask him
>> questions). If not, if you don't mind, I'll send my question to this list.
>> Then I will certainly be back here, seeking for answers :)
> Yes feel free to ask here.
> I updated the _v4l2 branch with the latest stuff from last weekend.  The
> code is in here
> https://libwebsockets.org/git/libwebsockets/tree/minimal-examples/ws-server/minimal-ws-server-v4l2?h=_v4l2
> The objects are defined in private.h, notice there is a #define at the
> minute called OWN_MOOF that is set, this bypasses libav mux stream
> generation (which does not succeed to stream to Chrome either) and

[RCS] I'm pretty sure I know why ... :) When I wrote spec for
embedding nagra drm in chrome, the intership we had for
coding the cipher plugin had the same kind of problem.
You might have been using some kind of internal API for
accessing the streaming (sorry I don't remember the name),
And this API get an answer from you the first time you are called.
This answer MUST not be OK, but something as ... Need more
computing/packet .. don't remember sorry. My hack time in chrome
is pretty old :) But I'll have a look at it and try to guide you through.

> instead takes the h.264 encoder packets and copies and strips the annex
> b / TS stuff into buffers in struct src_inst, and puts them back
> according to client state rather than source video state.  If it's not
> set, it uses libav streams to do the muxing.
> annex-b.c in there parses that and produces the avcC format blob from
> the SPS and PPS.
> boxes.c became a complete box generation implementation which maintains
> a stack of parents and handles their length automatically.  Because it
> became easier to do that than try to continue fiddling them by hand.
> transcode.c takes the MJPEG as it comes and passes it through libav* as
> it stands, which passes it through libx264 and produces an AVPacket
> combining annex-b with h.264
> v4l2.c handles setting up the stream as MJPEG, it wants to prefer native
> h.264 if available but we are not at that point yet.\
> minimal-ws-server.c has an important define DUMP_TO_FILE, if set, it
> will take a copy of the first video client data to a file /tmp/str.mp4.
>    You can use this to confirm the output is OK in mplayer / vlc etc as
> well as by file:/// that it can be played in the browsers.
> To test just install the libav* dev packages and build lws with
> LWS_WITH_MINIMAL_EXAMPLES=1, you will get a
> ./build/bin/lws-minimal-ws-server-v4l2 that you should run from the cwd
> ./minimal-examples/ws-server/minimal-ws-server-v4l2
> Then visit in your browser, it defaults to trying to
> stream /dev/video0.
> -Andy

[RCS] Ok, clone your code and will go deeper with it... Need
to understand what you are trying to do (even if I've got a tiny idea :)

Gimme a couple of hours ...

>> Thanks guys... and keep up good work !
>> Regards
>> Rémi
>> _______________________________________________
>> Libwebsockets mailing list
>> Libwebsockets at ml.libwebsockets.org
>> https://libwebsockets.org/mailman/listinfo/libwebsockets
-------------- next part --------------
A non-text attachment was scrubbed...
Name: publickey - remi at cohenscali.net - c67eba31.asc
Type: application/pgp-keys
Size: 705 bytes
Desc: not available
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20210514/39878ad0/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 294 bytes
Desc: OpenPGP digital signature
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20210514/39878ad0/attachment.sig>

More information about the Libwebsockets mailing list