[Libwebsockets] Help wanted: V4L2 / mp4 / h.264

Silas Parker skyhisi+libwebsockets at gmail.com
Fri Apr 30 16:39:31 CEST 2021


Hi Andy,

I can't offer anything specific, but in the past I've used Shaka Packager (
https://github.com/google/shaka-packager) for generating MPEG DASH streams
which is a standard way of delivering streaming video to browsers and other
devices.  You might be able to glean what Chrome is after by looking at the
MP4 files that it generates.

Thanks,
Silas

On Fri, 30 Apr 2021 at 14:45, Andy Green <andy at warmcat.com> wrote:

>
>
> On 4/19/21 6:41 AM, Andy Green wrote:
> >
> >
> > On 4/12/21 6:53 AM, Andy Green wrote:
> >> Hi -
> >>
> >> I spent the last couple of weekends on a minimal example that works
> >> with v4l2 and libav*.  What I have is in a couple of patches ahead of
> >> main here
> >>
> >> https://libwebsockets.org/git/libwebsockets/log?h=_v4l2
> >
> > I spent this last weekend on this as well... it's still not there, but
> > it's getting close I think.  I updated _v4l2 branch.
>
> After a lot of struggling around this can transcode live webcam MJPG ->
> mp4 + h.264 -> wss -> Media Source on Linux Firefox (yay) but not yet
> Chrome (boo).  It uses the multitail lws_ring stuff and retains the ftyp
> / moov boxes and reapplies them as clients join, so you can open n tabs
> or have n clients on the same stream OK.
>
> I tested it serving from two Linux desktop class devices and from an Rpi
> zero (using h264_omx accelerated encoding) over wifi.  For h264_omx, the
> example adapts the annex-b NALs to appear on every packet as required
> for ffox.
>
> The client JS extracts the avc1 mimetype from the mp4 moov box in the
> stream now, the JS is reordered to get that info from the ws mp4 before
> initializing the mediasource.
>
> Current caveats:
>
> 1) Only tries to stream h.264 video, doesn't try to do audio atm.  But
> it is getting data via libav* mux / , so it can be added.
>
> 2) Works on Linux ffox well, the same stream can be saved to a file and
> play in vlc and mplayer well too, but Chrome won't accept it as a video
> stream, it hangs up on it after one or two packets,
> chrome://media-internals only says:
>
>    Append: stream parsing failed. Data size=12860 append_window_start=0
> append_window_end=inf
>
> I read that chrome requires a tfdt box in moof->traf, I added code to
> modify the moof section to add it reporting the packet pts, but it
> didn't convince Chrome enough.  So that's the last big mystery.
>
> 3) Right now the JS approach is try to display the most recent frame for
> minimal latency, but that has no queue to fall back on if the frame is
> late.  There is a queue in the JS but it needs more regulation to, eg,
> take the approach to always keep 3s of video in the queue and accept the
> latency.  But these adaptations, like spool to a file, are relatively
> easy once it works on Chrome.
>
> I'd be delighted if anyone who already fought these kind of issues can
> give some advice or suggestions on what's missing for Chrome.
>
> -Andy
> _______________________________________________
> Libwebsockets mailing list
> Libwebsockets at ml.libwebsockets.org
> https://libwebsockets.org/mailman/listinfo/libwebsockets
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://libwebsockets.org/pipermail/libwebsockets/attachments/20210430/bccecb58/attachment.htm>


More information about the Libwebsockets mailing list