[Libwebsockets] Help wanted: V4L2 / mp4 / h.264

Andy Green andy at warmcat.com
Fri Apr 30 15:45:38 CEST 2021

On 4/19/21 6:41 AM, Andy Green wrote:
> On 4/12/21 6:53 AM, Andy Green wrote:
>> Hi -
>> I spent the last couple of weekends on a minimal example that works 
>> with v4l2 and libav*.  What I have is in a couple of patches ahead of 
>> main here
>> https://libwebsockets.org/git/libwebsockets/log?h=_v4l2
> I spent this last weekend on this as well... it's still not there, but 
> it's getting close I think.  I updated _v4l2 branch.

After a lot of struggling around this can transcode live webcam MJPG -> 
mp4 + h.264 -> wss -> Media Source on Linux Firefox (yay) but not yet 
Chrome (boo).  It uses the multitail lws_ring stuff and retains the ftyp 
/ moov boxes and reapplies them as clients join, so you can open n tabs 
or have n clients on the same stream OK.

I tested it serving from two Linux desktop class devices and from an Rpi 
zero (using h264_omx accelerated encoding) over wifi.  For h264_omx, the 
example adapts the annex-b NALs to appear on every packet as required 
for ffox.

The client JS extracts the avc1 mimetype from the mp4 moov box in the 
stream now, the JS is reordered to get that info from the ws mp4 before 
initializing the mediasource.

Current caveats:

1) Only tries to stream h.264 video, doesn't try to do audio atm.  But 
it is getting data via libav* mux / , so it can be added.

2) Works on Linux ffox well, the same stream can be saved to a file and 
play in vlc and mplayer well too, but Chrome won't accept it as a video 
stream, it hangs up on it after one or two packets, 
chrome://media-internals only says:

   Append: stream parsing failed. Data size=12860 append_window_start=0 

I read that chrome requires a tfdt box in moof->traf, I added code to 
modify the moof section to add it reporting the packet pts, but it 
didn't convince Chrome enough.  So that's the last big mystery.

3) Right now the JS approach is try to display the most recent frame for 
minimal latency, but that has no queue to fall back on if the frame is 
late.  There is a queue in the JS but it needs more regulation to, eg, 
take the approach to always keep 3s of video in the queue and accept the 
latency.  But these adaptations, like spool to a file, are relatively 
easy once it works on Chrome.

I'd be delighted if anyone who already fought these kind of issues can 
give some advice or suggestions on what's missing for Chrome.


More information about the Libwebsockets mailing list