Hey Lynne, Adding our conversation from IRC:
<libmilos-so> Hello Lynne, I'm Milos, contributor to https://whitebox.aero . We're looking for a FFmpeg developer for assistance with filter development, and Paul B Mahol gave us your contact as an expert in Vulkan development. We are building an SBC-based device which needs to run the v360 filter on it, however the CPU filter's performance is starting to be our bottleneck, and we're looking to explore a GPU-based approach to it. Paul started working on `v360_{opencl,vulkan}` filters a while back, but it wasn't finished, and the `libavfilter` has changed since, requiring more development to get them working. Here's a link to the mailing list's previous discussion: https://ffmpeg.org/pipermail/ffmpeg-devel/2024-June/330229.html
<libmilos-so> Our use case is to convert dual-fisheye videos to equirectangular in real time, we're basically only doing this for now: `v360=dfisheye:e:ih_fov=193:iv_fov=193`
<libmilos-so> Currently we are running it for a single video feed, but we're close to hitting resource limits. As we'd like to extend to multiple videos, we think that GPU-based acceleration would provide us the performance improvement we need. Considering your FFmpeg expertise, we were wondering if you'd be interested for contracting to help us get this implemented for our use case? Our mailing list is at https://ml.whitebox.aero, and if that'd be alright with you, I'd like to copy the contents of this conversation to the mailing list later. If you're interested in the contractual work, it'd be great if we could then continue the conversation on the mailing list to keep the team in the loop
<libmilos-so> Thanks!
<Lynne> sure, I'd be happy to help
<libmilos-so> Thanks! Could you give me your email address and I'll cc you on the thread?
<Lynne> the filter which paul wrote years ago needs quite a bit of work, since currently we're rewriting all vulkan filters to compile-time spir-v generation
<Lynne> sure, [redacted]
For context, we are taking H.264/H.265 dual-fisheye videos from Insta360 cameras, both from live feeds, and when recorded from file recordings, and processing them using FFmpeg to produce equirectangular videos which would then be suitable for playback in browsers. We are currently using Pannellum player for this, given our current equirectangular format. If there are more efficient options on the backend side to achieve the same effect, we're open to exploring those as well, although frontend clients would also be phones and tablets, so no heavy operations should take place on them. We are currently developing on RK3588 (on Orange Pi 5 Plus), which by the documentation supports OpenCL 2.2 and Vulkan 1.2. It has H.264 and H.265 hardware encoders and decoders, which we've tried using, but v360 filter ended up being the biggest bottleneck. Here's one example of how we're currently using this filter:
return [ "ffmpeg",# Decoder settings "-fflags","+genpts+discardcorrupt", "-flags","low_delay",
"-f","h264", "-i","pipe:0",# input from stdin "-r","30",
# Filter settings "-filter_complex", f"[0:v]scale=1440:720,v360=dfisheye:e:ih_fov=193:iv_fov=193[out]", "-map","[out]",
# Encoder settings "-c:v","libx264", "-preset","ultrafast", "-tune","zerolatency", "-pix_fmt","yuv420p", "-profile:v","main", "-level","3.1",
# Bitrate settings # This ensures that keyframes are pushed frequently enough to keep the # streaming latency low "-g","30",# keyframe interval "-keyint_min","30",# minimum keyframe interval "-sc_threshold","0",# disable scene change detection "-muxpreload","0", "-muxdelay","0",
"-f","flv",# output format "-rtmp_live","live",
# e.g.: rtmp://srs-dev/live/livestream output_target, ]
By my understanding, we likely need a GPU-based v360 filter as a drop-in replacement for the standard CPU-based v360. I think that the GPU-based filters would also require `hwupload`/`hwdownload`(?), but I don't have experience with it. Personally I'm new to FFmpeg, so feel free to correct me if something sounds off. The code would ideally be released under a copyleft license, I'll let Xavier pitch in on this. Does this sound like a proper approach to it and something that'd be interesting for you to work on? Thanks!