[Libwebsockets] Help wanted: V4L2 / mp4 / h.264

Rémi COHEN-SCALI ... remi at cohenscali.net
Fri May 14 15:31:35 CEST 2021


I wasn't subscribed with mailling list at this message writing time then
after reading archives i catch this opportunity to answer some of the
questions I found.
Answers inlined

> On 4/30/21 3:39 PM, Silas Parker wrote:
> >/Hi Andy, /> >//> >/I can't offer anything specific, but in the past I've used Shaka /> >/Packager (https://github.com/google/shaka-packager 
<https://github.com/google/shaka-packager> /> >/<https://github.com/google/shaka-packager 
<https://github.com/google/shaka-packager>>) for generating MPEG DASH /> >/streams which is a standard way of delivering streaming video to /> >/browsers and other devices.  You might be able to glean what Chrome is /> >/after by looking at the MP4 files that it generates. /> 
> Thanks for the hint Silas.
> I started looking at this earlier today, rather than compare output, 
> it's interesting to see what it says when I give it what's produced when 
> spooled to a /tmp file as its input.
> It complained I had disabled some mp4 boxes that were not needed on ffox 
> and errored out on each, so that's a significant clue.
> After it was OK at mp4 level, it complains now about finding a type 13 
> NAL at the NALU layer, I instrumented shaka to print the NAL type

As you certainly understand, bmff is made of boxes included the one in the
others. The outer one being the container itself and the inners going to groups
of macro blocks (or many other things). All these define some kind of database
you can access to get data, boxes specifying the model, the kind of object you
are dealing with.
Now the use cases include straeming, progressive download or random access to
this database. For the streaming use case (and in some extent for the progressive
download use case), the bit stream will have to be packetized. NAL (Network Access
Layer) and NALU (Network Access Layer Unit) are the result of this packetization.
Most boxes types are defined in BMFF. BMFF is the generic container (at the top
of the classe tree). It contains definitions for standard/most generic group of
data. Then some specializations exist for all derived use cases, mp4 file supports
random access, progressive download (or perhaps gp3 spec for this use case...
I don't remember) and stream.
Of course all these use cases are about video/audio compression, then every possible
way to save space is good. SPS and PPS stands for 'Sequence Parameter Set' and
'Picture Parameter Set'. They are boxes transporting set of parameters applicable to
some sequences or group of sequences (GOP structure, etc,)  and to some pictures
or group of pictures (more or less general parameters for algo as decoding, deblocking,
rendering). For allowing space saving. Some parameters are not repeated. As the boxes
are included in each others, some parameters can concern several inner boxes as they
are not redefined for their values to change.
At start of container, you have one SPS box and one PPS box and if parameters do not
need to be modified, you will not see them anymore.
A decoder maintain a database with some context in order to be able to correctly retrieve
which parameter must be applied to which part of the stream. In order to be able to rebuild
picture, the same problem occurs because the decoding order of the frames is not the
same as the playing order of the frame (because of inter & intra coding. inter, the frame
can reference another before her in the decoding order, but after her in the playing order).
However this is a little bit out of scope here :)
Hope I answered more questions than I add :)
Feel free to ask something more specific if you need.

> $ ./out/Release/packager 
> 'in=/tmp/str.mp4,stream=video,output=/tmp/x.mp4' 
> --generate_static_live_mpd --mpd_output h264.mpd
> Zoiper Click2Dial[0503/110045 <zoiper:0503110045>FR:INFO:demuxer.cc(89)] Demuxer::Run() on file '/tmp/str.mp4'.
> Zoiper Click2Dial[0503/110045 <zoiper:0503110045>FR:INFO:demuxer.cc(155)] Initialize Demuxer for file
> '/tmp/str.mp4'.
> Zoiper Click2Dial[0503/110045 <zoiper:0503110045>FR:ERROR:avc_decoder_configuration_record.cc(53)] 13
> Zoiper Click2Dial[0503/110045 <zoiper:0503110045>FR:ERROR:avc_decoder_configuration_record.cc(54)] Failure
> while processing: nalu.type() == Nalu::H264_SPS
> Zoiper Click2Dial[0503/110045 <zoiper:0503110045>FR:ERROR:mp4_media_parser.cc(605)] Failed to parse avcc.

[RCS] avcc is the code that provide the kind of compression used, here AVC (advanced video coding === h265)

> Zoiper Click2Dial[0503/110045 <zoiper:0503110045>FR:ERROR:packager_main.cc(550)] Packaging Error: 8
> (PARSER_FAILURE): Cannot parse media file /tmp/str.mp4
> but I can't see any type 13 NAL ("SPS extension") in the hexdump from a 
> quick look.  There's also some problem with slightly truncated h.264 
> stream according to mplayer

[RCS] There is always a SPS ate stream start.

> [mov,mp4,m4a,3gp,3g2,mj2 @ 0x7fe9bb5e6340]stream 0, sample 777, dts 25899974
> 32768=mp_read(0x55cffa2c4220, 0x55cffa2c9100, 32768), pos: 13043046, eof:0
> vd_ffmpeg data: 0000, bd05700f, 0000, 15160500
> [h264 @ 0x7fe9bad80040]nal_unit_type: 5(IDR), nal_ref_idc: 3
> [h264 @ 0x7fe9bad80040]nal_unit_type: 5(IDR), nal_ref_idc: 3
> [h264 @ 0x7fe9bad80040]nal_unit_type: 5(IDR), nal_ref_idc: 3
> [h264 @ 0x7fe9bad80040]nal_unit_type: 5(IDR), nal_ref_idc: 3
> [h264 @ 0x7fe9bad80040]nal_unit_type: 5(IDR), nal_ref_idc: 3
> [h264 @ 0x7fe9bad80040]bytestream overread -17
> [h264 @ 0x7fe9bad80040]error while decoding MB 39 22, bytestream -17
> [h264 @ 0x7fe9bad80040]concealing 50 DC, 50 AC, 50 MV errors in I frame
> V:  25.8   0/  0 15%  0%  0.0% 0 0
> ...it shows up as a thin area of corrupted decode at the very bottom right.
> Even so, I can play the output file now in Chrome itself using a 
 > file:///tmp/str.mp4  URL, it can't be a million miles away.
> So I will continue with this at the weekend, it's certainly better than 
> fiddling with things in the dark.
> -Andy

[RCS] I'm going to clone your branch and try to see what is the purpose of
bmff/avc manipulation you are trying to achieve.

Good luck


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20210514/724b6b81/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: publickey - remi at cohenscali.net - c67eba31.asc
Type: application/pgp-keys
Size: 705 bytes
Desc: not available
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20210514/724b6b81/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 294 bytes
Desc: OpenPGP digital signature
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20210514/724b6b81/attachment-0001.sig>

More information about the Libwebsockets mailing list