[Libwebsockets] Help wanted: V4L2 / mp4 / h.264

Andy Green andy at warmcat.com
Mon Apr 19 07:41:42 CEST 2021

On 4/12/21 6:53 AM, Andy Green wrote:
> Hi -
> I spent the last couple of weekends on a minimal example that works with 
> v4l2 and libav*.  What I have is in a couple of patches ahead of main here
> https://libwebsockets.org/git/libwebsockets/log?h=_v4l2

I spent this last weekend on this as well... it's still not there, but 
it's getting close I think.  I updated _v4l2 branch.

I assumed that libav* would wrap production of mp4 container for 
streaming like it does for producing files, via AVStream.  But as far as 
I could find, it doesn't, it just leaves that "as an exercise for the 
reader".  Attempting to wire it up to a custom AVIOContext that would 
stream the output was not possible with mp4 container, since it insists 
to have a seek callback that has no meaning for a socket-based stream.

Therefore AFAIK the code has to manually handle producing a streamable 
mp4 container layer, and replacing Annex B h.264 frame headers (suitable 
for MPEG-TS but not accepted by much else) with AVCC form at h.264 level.

The example produces both things, the mp4 container is a canned table 
for now.  By mirroring the stream to a file, I can see it can be parsed 
as mp4 by various things (alhough whether they can render them is 
another matter).  MediaSource rejects the mp4 header.

$ mediainfo /tmp/str.mp4
Complete name                            : /tmp/str.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (avc1)
File size                                : 543 KiB
Encoded date                             : UTC 1970-01-01 00:00:00
Tagged date                              : UTC 1970-01-01 00:00:00
IsTruncated                              : Yes

ID                                       : 2
Title                                    : lws stream
Language                                 : English
Encoded date                             : UTC 1970-01-01 00:00:00
Tagged date                              : UTC 1970-01-01 00:00:00


Checking for MOV
ISO: File Type Major Brand: ISO Base Media
ISO: File Type Minor Version: 0
ISO: File Type Compatible Brand #0: avc1
MOV: Movie header found!
MOV: Movie header: start: 1c end: 13f
MOV: unknown chunk: moof 56
MOV: Movie DATA found!
MOV: Movie data: start: 17f end: 1000176
Quicktime/MOV file format detected.
lschunks dhvm  100
MOV: Movie header (100 bytes): tscale=1000  dur=2147483647
lschunks kart  175
MOV: Track #0:
lschunks dhkt  84
MOV:  Track header!
tkhd len=84 ver=0 flags=0x0 id=2 dur=-1 lay=0 vol=0
lschunks aidm  75
MOV:  Media stream!
lschunks dhdm  24
MOV:   Media header!
lschunks rldh  35
MOV:   Handler header: /vide () ws stream
MOV: unknown handler class: 0x0 ()
MOV track #0: 0 chunks, 0 samples
pts=-1  scale=1000  time=4294967.500
*** constant samplesize & variable duration not yet supported! ***
Contact the author if you have such sample file!
Unknown track type found (type: 0)
MOV: longest streams: A: #-1 (0 samples)  V: #-1 (0 samples)
No stream found.


[00007fbce8c01d30] mp4 stream debug: dumping root Box "root"
[00007fbce8c01d30] mp4 stream debug: |   + ftyp size 20 offset 0
[00007fbce8c01d30] mp4 stream debug: |   + moov size 299 offset 20
[00007fbce8c01d30] mp4 stream debug: |   |   + mvhd size 108 offset 28
[00007fbce8c01d30] mp4 stream debug: |   |   + trak size 183 offset 136
[00007fbce8c01d30] mp4 stream debug: |   |   |   + tkhd size 92 offset 144
[00007fbce8c01d30] mp4 stream debug: |   |   |   + mdia size 83 offset 236
[00007fbce8c01d30] mp4 stream debug: |   |   |   |   + mdhd size 32 
offset 244
[00007fbce8c01d30] mp4 stream debug: |   |   |   |   + hdlr size 43 
offset 276
[00007fbce8c01d30] mp4 stream debug: |   + moof size 56 offset 319
[00007fbce8c01d30] mp4 stream debug: |   |   + traf size 48 offset 327
[00007fbce8c01d30] mp4 stream debug: |   |   |   + tfhd size 16 offset 335
[00007fbce8c01d30] mp4 stream debug: |   |   |   + trun size 24 offset 351
[00007fbce8c01d30] mp4 stream debug: |   + mdat size 589823 offset 375

mplayer can parse it, but it does not know what the moof box is.

At the moment the sticking point is while the players understand the mp4 
header and see the video track, something is missing at the mp4 layer so 
that they feel the video track should be played.  The corresponding 
moov->trak->tkhd box has the flags indicating it should be presented.


More information about the Libwebsockets mailing list