Re: Fwd: Re: Re: FFmpeg v360 Vulkan filter development
Hi Lynne, That sounds all good to me! Your rate and the 35 hours work - if you see that you might need more, let me know before reaching that point. It will in any case be a good occasion to see where we stand. If you would like to send an invoice for an initial deposit to start the work, that would be fine. For the drivers, it would be good to be able to run the newer drivers - even if we have to apply patches & recompile manually for now, as long as the code used is eventually being merged into the upstream mainline, it is fine as it will not be necessary to apply any newer changes manually once they have propagated. We have to make sure they do make it upstream, though. -- Xavier Whitebox.aero <https://whitebox.aero/> On Thu, Mar 26, 2026, at 14:40, Lynne wrote:
Hi,
I hope Milos have time to look into this. I would have no issues writing the filter, my hourly rate is 150 EUR, and I estimate it would take about 35 hours. I would be able to complete it by the week after the next, if you accept.
The concern I have is whether you would be able to switch the software to newer drivers, capable of handling the Vulkan code. The SoC ecosystem, which the OrangePi is a part of, has been moving to open-source drivers and mainline kernel versions. But these changes take time to propagate and enable by default, and in the case of the OrangePi, require extra steps after installation to activate.
The newer open-source drivers will soon be able to handle hardware decoding of the video to reduce even more CPU usage and latency. The Vulkan filter is capable of interoperability with hardware decoding, which would also eliminate the need to upload video frames. So I think that you should look into using the Mesa drivers to avoid being locked out of new features in the future.
On 24/03/2026 11:02, Xavier Antoviaque wrote:
Hi Lynne,
Thanks for your replies - Milos is off this week, in case he hasn't set an autoresponder. He will be back next week and look into this.
Btw if you want to start charging for the time you spend helping Milos, it would be fair.
-- Xavier Whitebox.aero <https://whitebox.aero/>
On Fri, Mar 20, 2026, at 08:59, Lynne wrote:
Those drivers are incredibly old. Did you have any chance running the panfrost drivers from Mesa?
On 20/03/2026 03:30, milos@whitebox.aero wrote:
Not sure what's going on with the emails, I didn't receive the original email, just got this reply. I've run it again with `format=yuv420p`, still getting practically the same result - created new file in the snippet named: `run with trace and format=yuv420p`:
https://gitlab.com/whitebox-aero/whitebox/-/snippets/5967510
I've tried looking at the logs but I'm not sure what I'm doing wrong. Here's also a diff between the first and second outputs if that helps, left one is the previous run, right one is this new run:
https://www.diffchecker.com/qPET78MB/
On 19. 3. 2026. 15:32, Xavier Antoviaque wrote:
Milos -- making sure you have received and seen this email reply from Lynne, below.
-- Xavier
On Tue, Mar 17, 2026, at 18:53, Lynne wrote:
Ah, ok. You need to add ",format=yuv420p" after hwdownload, e.g.: "hwupload,avgblur_vulkan=planes=0,hwdownload,format=yuv420p"
Could you test again?
On 17/03/2026 18:22, milos@whitebox.aero wrote: > Hey Lynne, > > I only got this email by Xavier, sorry. > > There seems to be a space inside the initial URL, maybe the email > client > mangled it somehow. I've reuploaded it to pastebin: https:// > pastebin.com/BjmkgkzL > > On 16. 3. 2026. 18:00, Xavier Antoviaque wrote: >> I did get it on my side, - Milos those emails are visible at https:// >> ml.whitebox.aero/archives/list/dev@ml.whitebox.aero/thread/ >> JYXZMGH7NMFPNXXNQ2KDUAGGIC6IPRZK/ (the email shown on that page, and >> the one that you can expand below it). >> >> And FYI, hopefully the mailing issue should be solved now, which >> allowed your email to show up there so I could link to it :) >> >> Btw Lynne did you get my previous email about how to contract? If it >> has been lost in the mess I can send it to you again. >> >> -- >> Xavier >> Whitebox.aero <https://whitebox.aero/> >> >> On Sat, Mar 14, 2026, at 00:28, Lynne wrote: >>> Err, didn't you receive my 2 replies? Today, and 2 days ago. >>> >>> On 14/03/2026 00:18, milos@whitebox.aero wrote: >>>> Hey Lynne, >>>> >>>> Just wanna check if you've seen our previous emails. >>>> >>>> Thanks! >>>> >>>> On 10. 3. 2026. 22:35, Milos Petkovic via whitebox-dev wrote: >>>>> Hey Lynne, >>>>> >>>>> Can't link directly to the file itself, but I've updated the >>>>> previously linked snippet, added a new file named `run with trace`: >>>>> https://gitlab.com/whitebox- aero/whitebox/-/snippets/5967510 >>>>> >>>>> I've never worked with Vulkan before so I kinda expected this >>>>> test to >>>>> be a drop in replacement, but looking into all this I feel I'm a >>>>> bit >>>>> far from understanding exactly how all these fit in, and being >>>>> confident I'm actually giving you accurate feedback information. >>>>> Would >>>>> it be alright to include these tests in the scope of your >>>>> contract as >>>>> well so we can get the development going faster? >>>>> >>>>> We are running on Orange Pi 5 Plus [1], and we're open for both you >>>>> sourcing the components yourself and invoicing it, or us arranging >>>>> shipping for you. We have hardware setup instructions [2] for >>>>> getting >>>>> started quickly if you'd want to start from our current environment >>>>> (the other listed components should not be needed to get >>>>> started, just >>>>> for full operation). >>>>> >>>>> Would you be interested in this? >>>>> >>>>> [1] http://www.orangepi.org/html/hardWare/ >>>>> computerAndMicrocontrollers/ >>>>> details/Orange-Pi-5-plus.html >>>>> >>>>> [2] https://docs.whitebox.aero/hardware_setup/ >>>>> >>>>> On 10. 3. 2026. 13:28, Xavier Antoviaque wrote: >>>>>> Hi Lynne, nice to meet you! >>>>>> >>>>>> Sorry it took me some time to get up to speed here, the mailing- >>>>>> list issues got in the way - we are still solving them btw, but >>>>>> you can mostly ignore them on your side, just keep Milos and my >>>>>> email in CC besides the ML and we should be fine :) >>>>>> >>>>>> And thanks for helping with this, anf for being interested in >>>>>> contracting on the topic. How would you like to setup the contract >>>>>> work on this? Generally we pay hourly, agreeing on a hourly rate >>>>>> and a rough estimation of the time it will take you - then if it >>>>>> looks like you might need more time, you check with me and we >>>>>> rediscuss? If you want to keep anything confidential (eg rate, >>>>>> personal info, etc.), you can also write to me directly >>>>>> (xavier@whitebox.aero) - just keep the non-confidential info in >>>>>> the public thread here so that others can follow the rest. >>>>>> >>>>>> Looking forward! >>>>>> >>>>>> Xavi. >>>>>> >>>>>> On Tue, Mar 10, 2026, at 11:14, Milos Petkovic via whitebox-dev >>>>>> wrote: >>>>>>> Just a heads up, it seems the ML issue is still there - >>>>>>> https://ml.whitebox.aero/archives/list/dev@ml.whitebox.aero/ >>>>>>> thread/ >>>>>>> RE7TNLBLOSH6CC5HAJFZ2E4B6UZT32AL/, >>>>>>> so this reply will end up on the ML but probably in a separate >>>>>>> (new) >>>>>>> thread. >>>>>>> >>>>>>> Thanks, I'll get back with the results when I try it out >>>>>>> >>>>>>> On 9. 3. 2026. 23:43, Lynne wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Thanks for sorting the email issue out. >>>>>>>> >>>>>>>> I think you should definitely rerun the test on 6.13 (I presume >>>>>>>> this >>>>>>>> uses the mesa panfrost/panvk driver) and use the latest >>>>>>>> FFmpeg git >>>>>>>> master. 8.1 which will ship next week has a ton of improvements. >>>>>>>> >>>>>>>> Just in case, could you post the log by adding -trace at the >>>>>>>> start? >>>>>>>> >>>>>>>> On 09/03/2026 22:26,milos@whitebox.aero wrote: >>>>>>>>> Hey Lynne, >>>>>>>>> >>>>>>>>> Your emails should go through instantly now. >>>>>>>>> >>>>>>>>> I've tried running that filter, but I couldn't get it working, >>>>>>>>> neither as part of our current pipeline, nor standalone (using >>>>>>>>> only >>>>>>>>> the filter from your sample). Pasted my results here: >>>>>>>>> https://gitlab.com/whitebox- aero/whitebox/-/snippets/5967510 >>>>>>>>> >>>>>>>>> I see that Vulkan has had considerable driver improvements in >>>>>>>>> kernel >>>>>>>>> 6.13, and we're currently running on 6.1 - would upgrading be a >>>>>>>>> requirement to running this (whether Vulkan, or the latest >>>>>>>>> FFmpeg >>>>>>>>> version in some way)? I'll try to get up to speed with the >>>>>>>>> driver >>>>>>>>> replacements later this week to verify that as well. >>>>>>>>> >>>>>>>>> @Xavier @Avinash Does any of this ring any bells, have you >>>>>>>>> maybe >>>>>>>>> tried using Vulkan on OPi before? It seems there's also Armbian >>>>>>>>> for >>>>>>>>> Orange Pi 5 Plus, but they ship it with 6.1 and 6.11, so I >>>>>>>>> guess that >>>>>>>>> wouldn't really bring benefits here >>>>>>>>> >>>>>>>>> On 6. 3. 2026. 16:43,milos@whitebox.aero wrote: >>>>>>>>>> Hey, I did receive it on my email, so I assume the ML also got >>>>>>>>>> it, >>>>>>>>>> but I think it requires approval to be posted on there. I >>>>>>>>>> don't have >>>>>>>>>> access to that, so when Xavier comes around he'll do it. I've >>>>>>>>>> sent >>>>>>>>>> him an email. I could technically just reply to your email >>>>>>>>>> here, but >>>>>>>>>> I think that'd open a new thread. >>>>>>>>>> >>>>>>>>>> Also thank you for the test instructions! I'll run them >>>>>>>>>> over the >>>>>>>>>> weekend and let you know how it went >>>>>>>>>> >>>>>>>>>> On 6. 3. 2026. 14:51, Lynne wrote: >>>>>>>>>>> Was my reply received? I don't see it on the ML. >>>>>>>>>>> >>>>>>>>>>> On 06/03/2026 13:28, Lynne wrote: >>>>>>>>>>>> Hi all, >>>>>>>>>>>> >>>>>>>>>>>> Certainly, a Vulkan version would likely speed the >>>>>>>>>>>> pipeline up. >>>>>>>>>>>> GPUs are good at interpolating, and very large amount of the >>>>>>>>>>>> time >>>>>>>>>>>> spent on the CPU version of the filter is spent doing that. >>>>>>>>>>>> >>>>>>>>>>>> Would you be able to test whether Vulkan is setup correctly? >>>>>>>>>>>> Try >>>>>>>>>>>> adding: >>>>>>>>>>>> `hwupload,avgblur_vulkan=planes=0,hwdownload,format=yuv420p` >>>>>>>>>>>> after the v360 filter. The output should not change, and the >>>>>>>>>>>> slowdown shouldn't be too significant. >>>>>>>>>>>> You need to use current git master of FFmpeg, not any >>>>>>>>>>>> release branch. >>>>>>>>>>>> >>>>>>>>>>>> I highly suggest using the panfrost Vulkan drivers from >>>>>>>>>>>> Mesa, >>>>>>>>>>>> rather than some outdated vendor-provided inactively >>>>>>>>>>>> developed >>>>>>>>>>>> drivers. They support RK3588, but you'll need to look into >>>>>>>>>>>> how to >>>>>>>>>>>> enable them (they require using the mainline kernel panfrost >>>>>>>>>>>> module, rather than mali). >>>>>>>>>>>> >>>>>>>>>>>> The previous work done by Paul isn't really helpful, as >>>>>>>>>>>> we're >>>>>>>>>>>> currently rewriting all the Vulkan codecs and filters to >>>>>>>>>>>> compile- >>>>>>>>>>>> time SPIR-V in order to eliminate dependencies on large >>>>>>>>>>>> external >>>>>>>>>>>> libraries, and to speed up the code with offline >>>>>>>>>>>> optimizations. >>>>>>>>>>>> I'd be interested in being contracted to write a modern >>>>>>>>>>>> Vulkan >>>>>>>>>>>> filter with enough capabilities for your particular use- >>>>>>>>>>>> case, if >>>>>>>>>>>> you agree. >>>>>>>>>>>> >>>>>>>>>>>> On 06/03/2026 12:37,milos@whitebox.aero wrote: >>>>>>>>>>>>> Hey Lynne, >>>>>>>>>>>>> >>>>>>>>>>>>> Adding our conversation from IRC: >>>>>>>>>>>>> >>>>>>>>>>>>>> <libmilos-so> Hello Lynne, I'm Milos, contributor to >>>>>>>>>>>>>> https:// >>>>>>>>>>>>>> whitebox.aero . We're looking for a FFmpeg developer for >>>>>>>>>>>>>> assistance with filter development, and Paul B Mahol >>>>>>>>>>>>>> gave us >>>>>>>>>>>>>> your contact as an expert in Vulkan development. We are >>>>>>>>>>>>>> building >>>>>>>>>>>>>> an SBC-based device which needs to run the v360 filter >>>>>>>>>>>>>> on it, >>>>>>>>>>>>>> however the CPU filter's performance is starting to be our >>>>>>>>>>>>>> bottleneck, and we're looking to explore a GPU-based >>>>>>>>>>>>>> approach to >>>>>>>>>>>>>> it. Paul started working on `v360_{opencl,vulkan}` >>>>>>>>>>>>>> filters a >>>>>>>>>>>>>> while back, but it wasn't finished, and the >>>>>>>>>>>>>> `libavfilter` has >>>>>>>>>>>>>> changed since, requiring more development to get them >>>>>>>>>>>>>> working. >>>>>>>>>>>>>> Here's a link to the mailing list's previous discussion: >>>>>>>>>>>>>> https://ffmpeg.org/ pipermail/ffmpeg-devel/2024- >>>>>>>>>>>>>> June/330229.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> <libmilos-so> Our use case is to convert dual-fisheye >>>>>>>>>>>>>> videos to >>>>>>>>>>>>>> equirectangular in real time, we're basically >>>>>>>>>>>>>> only doing this >>>>>>>>>>>>>> for now: `v360=dfisheye:e:ih_fov=193:iv_fov=193` >>>>>>>>>>>>>> >>>>>>>>>>>>>> <libmilos-so> Currently we are running it for a single >>>>>>>>>>>>>> video >>>>>>>>>>>>>> feed, but we're close to hitting resource limits. As we'd >>>>>>>>>>>>>> like >>>>>>>>>>>>>> to extend to multiple videos, we think that GPU-based >>>>>>>>>>>>>> acceleration would provide us the performance >>>>>>>>>>>>>> improvement we >>>>>>>>>>>>>> need. Considering your FFmpeg expertise, we were >>>>>>>>>>>>>> wondering if >>>>>>>>>>>>>> you'd be interested for contracting to help us get this >>>>>>>>>>>>>> implemented for our use case? >>>>>>>>>>>>>> Our mailing list is athttps://ml.whitebox.aero, and if >>>>>>>>>>>>>> that'd >>>>>>>>>>>>>> be alright with you, I'd like to copy the contents of this >>>>>>>>>>>>>> conversation to the mailing list later. If you're >>>>>>>>>>>>>> interested in >>>>>>>>>>>>>> the contractual work, it'd be great if we could then >>>>>>>>>>>>>> continue >>>>>>>>>>>>>> the conversation on the mailing list to keep the team in >>>>>>>>>>>>>> the loop >>>>>>>>>>>>>> >>>>>>>>>>>>>> <libmilos-so> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> <Lynne> sure, I'd be happy to help >>>>>>>>>>>>>> >>>>>>>>>>>>>> <libmilos-so> Thanks! Could you give me your email address >>>>>>>>>>>>>> and >>>>>>>>>>>>>> I'll cc you on the thread? >>>>>>>>>>>>>> >>>>>>>>>>>>>> <Lynne> the filter which paul wrote years ago needs quite >>>>>>>>>>>>>> a bit >>>>>>>>>>>>>> of work, since currently we're rewriting all vulkan >>>>>>>>>>>>>> filters to >>>>>>>>>>>>>> compile- time spir-v generation >>>>>>>>>>>>>> >>>>>>>>>>>>>> <Lynne> sure, [redacted] >>>>>>>>>>>>>> >>>>>>>>>>>>> For context, we are taking H.264/H.265 dual-fisheye videos >>>>>>>>>>>>> from >>>>>>>>>>>>> Insta360 cameras, both from live feeds, and when >>>>>>>>>>>>> recorded from >>>>>>>>>>>>> file recordings, and processing them using FFmpeg to >>>>>>>>>>>>> produce >>>>>>>>>>>>> equirectangular videos which would then be suitable for >>>>>>>>>>>>> playback >>>>>>>>>>>>> in browsers. We are currently using Pannellum player for >>>>>>>>>>>>> this, >>>>>>>>>>>>> given our current equirectangular format. If there are more >>>>>>>>>>>>> efficient options on the backend side to achieve the same >>>>>>>>>>>>> effect, >>>>>>>>>>>>> we're open to exploring those as well, although frontend >>>>>>>>>>>>> clients >>>>>>>>>>>>> would also be phones and tablets, so no heavy operations >>>>>>>>>>>>> should >>>>>>>>>>>>> take place on them. >>>>>>>>>>>>> >>>>>>>>>>>>> We are currently developing on RK3588 (on Orange Pi 5 >>>>>>>>>>>>> Plus), >>>>>>>>>>>>> which by the documentation supports OpenCL 2.2 and Vulkan >>>>>>>>>>>>> 1.2. It >>>>>>>>>>>>> has H.264 and H.265 hardware encoders and decoders, which >>>>>>>>>>>>> we've >>>>>>>>>>>>> tried using, but v360 filter ended up being the biggest >>>>>>>>>>>>> bottleneck. >>>>>>>>>>>>> >>>>>>>>>>>>> Here's one example of how we're currently using this >>>>>>>>>>>>> filter: >>>>>>>>>>>>> >>>>>>>>>>>>>> return [ >>>>>>>>>>>>>> "ffmpeg",# Decoder settings >>>>>>>>>>>>>> "-fflags","+genpts+discardcorrupt", >>>>>>>>>>>>>> "-flags","low_delay", >>>>>>>>>>>>>> >>>>>>>>>>>>>> "-f","h264", >>>>>>>>>>>>>> "-i","pipe:0",# input from stdin "-r","30", >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Filter settings "-filter_complex", >>>>>>>>>>>>>> f"[0:v]scale=1440:720,v360=dfisheye:e:ih_fov=193:iv_fov=193[out]", >>>>>>>>>>>>>> "-map","[out]", >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Encoder settings "-c:v","libx264", >>>>>>>>>>>>>> "-preset","ultrafast", >>>>>>>>>>>>>> "-tune","zerolatency", >>>>>>>>>>>>>> "-pix_fmt","yuv420p", >>>>>>>>>>>>>> "-profile:v","main", >>>>>>>>>>>>>> "-level","3.1", >>>>>>>>>>>>>> >>>>>>>>>>>>>> # Bitrate settings # This ensures that keyframes >>>>>>>>>>>>>> are pushed >>>>>>>>>>>>>> frequently enough to keep the # streaming latency low >>>>>>>>>>>>>> "-g","30",# keyframe interval "-keyint_min","30",# minimum >>>>>>>>>>>>>> keyframe interval "- sc_threshold","0",# disable scene >>>>>>>>>>>>>> change >>>>>>>>>>>>>> detection "- muxpreload","0", >>>>>>>>>>>>>> "-muxdelay","0", >>>>>>>>>>>>>> >>>>>>>>>>>>>> "-f","flv",# output format "-rtmp_live","live", >>>>>>>>>>>>>> >>>>>>>>>>>>>> # e.g.:rtmp://srs-dev/live/livestream >>>>>>>>>>>>>> output_target, >>>>>>>>>>>>>> ] >>>>>>>>>>>>> By my understanding, we likely need a GPU-based v360 filter >>>>>>>>>>>>> as a >>>>>>>>>>>>> drop- in replacement for the standard CPU-based v360. I >>>>>>>>>>>>> think >>>>>>>>>>>>> that the GPU- based filters would also require `hwupload`/ >>>>>>>>>>>>> `hwdownload`(?), but I don't have experience with it. >>>>>>>>>>>>> Personally >>>>>>>>>>>>> I'm new to FFmpeg, so feel free to correct me if something >>>>>>>>>>>>> sounds >>>>>>>>>>>>> off. >>>>>>>>>>>>> >>>>>>>>>>>>> The code would ideally be released under a copyleft >>>>>>>>>>>>> license, I'll >>>>>>>>>>>>> let Xavier pitch in on this. >>>>>>>>>>>>> >>>>>>>>>>>>> Does this sound like a proper approach to it and something >>>>>>>>>>>>> that'd >>>>>>>>>>>>> be interesting for you to work on? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks! >>>>>>>>>>>>> >>>>>>> _______________________________________________ >>>>>>> whitebox-dev mailing list --dev@ml.whitebox.aero >>>>>>> To unsubscribe send an email todev-leave@ml.whitebox.aero >>>>>> -- >>>>>> Xavier >>>>>> Whitebox.aero<https://whitebox.aero/> >>>>> _______________________________________________ >>>>> whitebox-dev mailing list --dev@ml.whitebox.aero >>>>> To unsubscribe send an email todev-leave@ml.whitebox.aero
Hey, Sorry for the late reply Lynne, I still don't get your emails for some reason :/ I've raised a ticket with my email provider, and will be migrating off of it if they don't fix it this week (and I guess I should migrate anyways regardless, they've been getting worse lately but this situation now is absurd). I haven't tinkered with the driver versions yet, will try that out this week and write back. Thanks for the help! On 30. 3. 2026. 16:31, Xavier Antoviaque wrote:
Hi Lynne,
That sounds all good to me! Your rate and the 35 hours work - if you see that you might need more, let me know before reaching that point. It will in any case be a good occasion to see where we stand. If you would like to send an invoice for an initial deposit to start the work, that would be fine.
For the drivers, it would be good to be able to run the newer drivers - even if we have to apply patches & recompile manually for now, as long as the code used is eventually being merged into the upstream mainline, it is fine as it will not be necessary to apply any newer changes manually once they have propagated. We have to make sure they do make it upstream, though.
-- Xavier Whitebox.aero <https://whitebox.aero/>
On Thu, Mar 26, 2026, at 14:40, Lynne wrote:
Hi,
I hope Milos have time to look into this. I would have no issues writing the filter, my hourly rate is 150 EUR, and I estimate it would take about 35 hours. I would be able to complete it by the week after the next, if you accept.
The concern I have is whether you would be able to switch the software to newer drivers, capable of handling the Vulkan code. The SoC ecosystem, which the OrangePi is a part of, has been moving to open-source drivers and mainline kernel versions. But these changes take time to propagate and enable by default, and in the case of the OrangePi, require extra steps after installation to activate.
The newer open-source drivers will soon be able to handle hardware decoding of the video to reduce even more CPU usage and latency. The Vulkan filter is capable of interoperability with hardware decoding, which would also eliminate the need to upload video frames. So I think that you should look into using the Mesa drivers to avoid being locked out of new features in the future.
On 24/03/2026 11:02, Xavier Antoviaque wrote:
Hi Lynne,
Thanks for your replies - Milos is off this week, in case he hasn't set an autoresponder. He will be back next week and look into this.
Btw if you want to start charging for the time you spend helping Milos, it would be fair.
-- Xavier Whitebox.aero <https://whitebox.aero/>
On Fri, Mar 20, 2026, at 08:59, Lynne wrote:
Those drivers are incredibly old. Did you have any chance running the panfrost drivers from Mesa?
On 20/03/2026 03:30, milos@whitebox.aero wrote:
Not sure what's going on with the emails, I didn't receive the original email, just got this reply. I've run it again with `format=yuv420p`, still getting practically the same result - created new file in the snippet named: `run with trace and format=yuv420p`:
https://gitlab.com/whitebox-aero/whitebox/-/snippets/5967510
I've tried looking at the logs but I'm not sure what I'm doing wrong. Here's also a diff between the first and second outputs if that helps, left one is the previous run, right one is this new run:
https://www.diffchecker.com/qPET78MB/
On 19. 3. 2026. 15:32, Xavier Antoviaque wrote:
Milos -- making sure you have received and seen this email reply from Lynne, below.
-- Xavier
On Tue, Mar 17, 2026, at 18:53, Lynne wrote: > Ah, ok. You need to add ",format=yuv420p" after hwdownload, e.g.: > "hwupload,avgblur_vulkan=planes=0,hwdownload,format=yuv420p" > > Could you test again? > > On 17/03/2026 18:22, milos@whitebox.aero wrote: >> Hey Lynne, >> >> I only got this email by Xavier, sorry. >> >> There seems to be a space inside the initial URL, maybe the email >> client >> mangled it somehow. I've reuploaded it to pastebin: https:// >> pastebin.com/BjmkgkzL >> >> On 16. 3. 2026. 18:00, Xavier Antoviaque wrote: >>> I did get it on my side, - Milos those emails are visible at https:// >>> ml.whitebox.aero/archives/list/dev@ml.whitebox.aero/thread/ >>> JYXZMGH7NMFPNXXNQ2KDUAGGIC6IPRZK/ (the email shown on that page, and >>> the one that you can expand below it). >>> >>> And FYI, hopefully the mailing issue should be solved now, which >>> allowed your email to show up there so I could link to it :) >>> >>> Btw Lynne did you get my previous email about how to contract? If it >>> has been lost in the mess I can send it to you again. >>> >>> -- >>> Xavier >>> Whitebox.aero <https://whitebox.aero/> >>> >>> On Sat, Mar 14, 2026, at 00:28, Lynne wrote: >>>> Err, didn't you receive my 2 replies? Today, and 2 days ago. >>>> >>>> On 14/03/2026 00:18, milos@whitebox.aero wrote: >>>>> Hey Lynne, >>>>> >>>>> Just wanna check if you've seen our previous emails. >>>>> >>>>> Thanks! >>>>> >>>>> On 10. 3. 2026. 22:35, Milos Petkovic via whitebox-dev wrote: >>>>>> Hey Lynne, >>>>>> >>>>>> Can't link directly to the file itself, but I've updated the >>>>>> previously linked snippet, added a new file named `run with trace`: >>>>>> https://gitlab.com/whitebox- aero/whitebox/-/snippets/5967510 >>>>>> >>>>>> I've never worked with Vulkan before so I kinda expected this >>>>>> test to >>>>>> be a drop in replacement, but looking into all this I feel I'm a >>>>>> bit >>>>>> far from understanding exactly how all these fit in, and being >>>>>> confident I'm actually giving you accurate feedback information. >>>>>> Would >>>>>> it be alright to include these tests in the scope of your >>>>>> contract as >>>>>> well so we can get the development going faster? >>>>>> >>>>>> We are running on Orange Pi 5 Plus [1], and we're open for both you >>>>>> sourcing the components yourself and invoicing it, or us arranging >>>>>> shipping for you. We have hardware setup instructions [2] for >>>>>> getting >>>>>> started quickly if you'd want to start from our current environment >>>>>> (the other listed components should not be needed to get >>>>>> started, just >>>>>> for full operation). >>>>>> >>>>>> Would you be interested in this? >>>>>> >>>>>> [1] http://www.orangepi.org/html/hardWare/ >>>>>> computerAndMicrocontrollers/ >>>>>> details/Orange-Pi-5-plus.html >>>>>> >>>>>> [2] https://docs.whitebox.aero/hardware_setup/ >>>>>> >>>>>> On 10. 3. 2026. 13:28, Xavier Antoviaque wrote: >>>>>>> Hi Lynne, nice to meet you! >>>>>>> >>>>>>> Sorry it took me some time to get up to speed here, the mailing- >>>>>>> list issues got in the way - we are still solving them btw, but >>>>>>> you can mostly ignore them on your side, just keep Milos and my >>>>>>> email in CC besides the ML and we should be fine :) >>>>>>> >>>>>>> And thanks for helping with this, anf for being interested in >>>>>>> contracting on the topic. How would you like to setup the contract >>>>>>> work on this? Generally we pay hourly, agreeing on a hourly rate >>>>>>> and a rough estimation of the time it will take you - then if it >>>>>>> looks like you might need more time, you check with me and we >>>>>>> rediscuss? If you want to keep anything confidential (eg rate, >>>>>>> personal info, etc.), you can also write to me directly >>>>>>> (xavier@whitebox.aero) - just keep the non-confidential info in >>>>>>> the public thread here so that others can follow the rest. >>>>>>> >>>>>>> Looking forward! >>>>>>> >>>>>>> Xavi. >>>>>>> >>>>>>> On Tue, Mar 10, 2026, at 11:14, Milos Petkovic via whitebox-dev >>>>>>> wrote: >>>>>>>> Just a heads up, it seems the ML issue is still there - >>>>>>>> https://ml.whitebox.aero/archives/list/dev@ml.whitebox.aero/ >>>>>>>> thread/ >>>>>>>> RE7TNLBLOSH6CC5HAJFZ2E4B6UZT32AL/, >>>>>>>> so this reply will end up on the ML but probably in a separate >>>>>>>> (new) >>>>>>>> thread. >>>>>>>> >>>>>>>> Thanks, I'll get back with the results when I try it out >>>>>>>> >>>>>>>> On 9. 3. 2026. 23:43, Lynne wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Thanks for sorting the email issue out. >>>>>>>>> >>>>>>>>> I think you should definitely rerun the test on 6.13 (I presume >>>>>>>>> this >>>>>>>>> uses the mesa panfrost/panvk driver) and use the latest >>>>>>>>> FFmpeg git >>>>>>>>> master. 8.1 which will ship next week has a ton of improvements. >>>>>>>>> >>>>>>>>> Just in case, could you post the log by adding -trace at the >>>>>>>>> start? >>>>>>>>> >>>>>>>>> On 09/03/2026 22:26,milos@whitebox.aero wrote: >>>>>>>>>> Hey Lynne, >>>>>>>>>> >>>>>>>>>> Your emails should go through instantly now. >>>>>>>>>> >>>>>>>>>> I've tried running that filter, but I couldn't get it working, >>>>>>>>>> neither as part of our current pipeline, nor standalone (using >>>>>>>>>> only >>>>>>>>>> the filter from your sample). Pasted my results here: >>>>>>>>>> https://gitlab.com/whitebox- aero/whitebox/-/snippets/5967510 >>>>>>>>>> >>>>>>>>>> I see that Vulkan has had considerable driver improvements in >>>>>>>>>> kernel >>>>>>>>>> 6.13, and we're currently running on 6.1 - would upgrading be a >>>>>>>>>> requirement to running this (whether Vulkan, or the latest >>>>>>>>>> FFmpeg >>>>>>>>>> version in some way)? I'll try to get up to speed with the >>>>>>>>>> driver >>>>>>>>>> replacements later this week to verify that as well. >>>>>>>>>> >>>>>>>>>> @Xavier @Avinash Does any of this ring any bells, have you >>>>>>>>>> maybe >>>>>>>>>> tried using Vulkan on OPi before? It seems there's also Armbian >>>>>>>>>> for >>>>>>>>>> Orange Pi 5 Plus, but they ship it with 6.1 and 6.11, so I >>>>>>>>>> guess that >>>>>>>>>> wouldn't really bring benefits here >>>>>>>>>> >>>>>>>>>> On 6. 3. 2026. 16:43,milos@whitebox.aero wrote: >>>>>>>>>>> Hey, I did receive it on my email, so I assume the ML also got >>>>>>>>>>> it, >>>>>>>>>>> but I think it requires approval to be posted on there. I >>>>>>>>>>> don't have >>>>>>>>>>> access to that, so when Xavier comes around he'll do it. I've >>>>>>>>>>> sent >>>>>>>>>>> him an email. I could technically just reply to your email >>>>>>>>>>> here, but >>>>>>>>>>> I think that'd open a new thread. >>>>>>>>>>> >>>>>>>>>>> Also thank you for the test instructions! I'll run them >>>>>>>>>>> over the >>>>>>>>>>> weekend and let you know how it went >>>>>>>>>>> >>>>>>>>>>> On 6. 3. 2026. 14:51, Lynne wrote: >>>>>>>>>>>> Was my reply received? I don't see it on the ML. >>>>>>>>>>>> >>>>>>>>>>>> On 06/03/2026 13:28, Lynne wrote: >>>>>>>>>>>>> Hi all, >>>>>>>>>>>>> >>>>>>>>>>>>> Certainly, a Vulkan version would likely speed the >>>>>>>>>>>>> pipeline up. >>>>>>>>>>>>> GPUs are good at interpolating, and very large amount of the >>>>>>>>>>>>> time >>>>>>>>>>>>> spent on the CPU version of the filter is spent doing that. >>>>>>>>>>>>> >>>>>>>>>>>>> Would you be able to test whether Vulkan is setup correctly? >>>>>>>>>>>>> Try >>>>>>>>>>>>> adding: >>>>>>>>>>>>> `hwupload,avgblur_vulkan=planes=0,hwdownload,format=yuv420p` >>>>>>>>>>>>> after the v360 filter. The output should not change, and the >>>>>>>>>>>>> slowdown shouldn't be too significant. >>>>>>>>>>>>> You need to use current git master of FFmpeg, not any >>>>>>>>>>>>> release branch. >>>>>>>>>>>>> >>>>>>>>>>>>> I highly suggest using the panfrost Vulkan drivers from >>>>>>>>>>>>> Mesa, >>>>>>>>>>>>> rather than some outdated vendor-provided inactively >>>>>>>>>>>>> developed >>>>>>>>>>>>> drivers. They support RK3588, but you'll need to look into >>>>>>>>>>>>> how to >>>>>>>>>>>>> enable them (they require using the mainline kernel panfrost >>>>>>>>>>>>> module, rather than mali). >>>>>>>>>>>>> >>>>>>>>>>>>> The previous work done by Paul isn't really helpful, as >>>>>>>>>>>>> we're >>>>>>>>>>>>> currently rewriting all the Vulkan codecs and filters to >>>>>>>>>>>>> compile- >>>>>>>>>>>>> time SPIR-V in order to eliminate dependencies on large >>>>>>>>>>>>> external >>>>>>>>>>>>> libraries, and to speed up the code with offline >>>>>>>>>>>>> optimizations. >>>>>>>>>>>>> I'd be interested in being contracted to write a modern >>>>>>>>>>>>> Vulkan >>>>>>>>>>>>> filter with enough capabilities for your particular use- >>>>>>>>>>>>> case, if >>>>>>>>>>>>> you agree. >>>>>>>>>>>>> >>>>>>>>>>>>> On 06/03/2026 12:37,milos@whitebox.aero wrote: >>>>>>>>>>>>>> Hey Lynne, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Adding our conversation from IRC: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> <libmilos-so> Hello Lynne, I'm Milos, contributor to >>>>>>>>>>>>>>> https:// >>>>>>>>>>>>>>> whitebox.aero . We're looking for a FFmpeg developer for >>>>>>>>>>>>>>> assistance with filter development, and Paul B Mahol >>>>>>>>>>>>>>> gave us >>>>>>>>>>>>>>> your contact as an expert in Vulkan development. We are >>>>>>>>>>>>>>> building >>>>>>>>>>>>>>> an SBC-based device which needs to run the v360 filter >>>>>>>>>>>>>>> on it, >>>>>>>>>>>>>>> however the CPU filter's performance is starting to be our >>>>>>>>>>>>>>> bottleneck, and we're looking to explore a GPU-based >>>>>>>>>>>>>>> approach to >>>>>>>>>>>>>>> it. Paul started working on `v360_{opencl,vulkan}` >>>>>>>>>>>>>>> filters a >>>>>>>>>>>>>>> while back, but it wasn't finished, and the >>>>>>>>>>>>>>> `libavfilter` has >>>>>>>>>>>>>>> changed since, requiring more development to get them >>>>>>>>>>>>>>> working. >>>>>>>>>>>>>>> Here's a link to the mailing list's previous discussion: >>>>>>>>>>>>>>> https://ffmpeg.org/ pipermail/ffmpeg-devel/2024- >>>>>>>>>>>>>>> June/330229.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <libmilos-so> Our use case is to convert dual-fisheye >>>>>>>>>>>>>>> videos to >>>>>>>>>>>>>>> equirectangular in real time, we're basically >>>>>>>>>>>>>>> only doing this >>>>>>>>>>>>>>> for now: `v360=dfisheye:e:ih_fov=193:iv_fov=193` >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <libmilos-so> Currently we are running it for a single >>>>>>>>>>>>>>> video >>>>>>>>>>>>>>> feed, but we're close to hitting resource limits. As we'd >>>>>>>>>>>>>>> like >>>>>>>>>>>>>>> to extend to multiple videos, we think that GPU-based >>>>>>>>>>>>>>> acceleration would provide us the performance >>>>>>>>>>>>>>> improvement we >>>>>>>>>>>>>>> need. Considering your FFmpeg expertise, we were >>>>>>>>>>>>>>> wondering if >>>>>>>>>>>>>>> you'd be interested for contracting to help us get this >>>>>>>>>>>>>>> implemented for our use case? >>>>>>>>>>>>>>> Our mailing list is athttps://ml.whitebox.aero, and if >>>>>>>>>>>>>>> that'd >>>>>>>>>>>>>>> be alright with you, I'd like to copy the contents of this >>>>>>>>>>>>>>> conversation to the mailing list later. If you're >>>>>>>>>>>>>>> interested in >>>>>>>>>>>>>>> the contractual work, it'd be great if we could then >>>>>>>>>>>>>>> continue >>>>>>>>>>>>>>> the conversation on the mailing list to keep the team in >>>>>>>>>>>>>>> the loop >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <libmilos-so> Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <Lynne> sure, I'd be happy to help >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <libmilos-so> Thanks! Could you give me your email address >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> I'll cc you on the thread? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <Lynne> the filter which paul wrote years ago needs quite >>>>>>>>>>>>>>> a bit >>>>>>>>>>>>>>> of work, since currently we're rewriting all vulkan >>>>>>>>>>>>>>> filters to >>>>>>>>>>>>>>> compile- time spir-v generation >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <Lynne> sure, [redacted] >>>>>>>>>>>>>>> >>>>>>>>>>>>>> For context, we are taking H.264/H.265 dual-fisheye videos >>>>>>>>>>>>>> from >>>>>>>>>>>>>> Insta360 cameras, both from live feeds, and when >>>>>>>>>>>>>> recorded from >>>>>>>>>>>>>> file recordings, and processing them using FFmpeg to >>>>>>>>>>>>>> produce >>>>>>>>>>>>>> equirectangular videos which would then be suitable for >>>>>>>>>>>>>> playback >>>>>>>>>>>>>> in browsers. We are currently using Pannellum player for >>>>>>>>>>>>>> this, >>>>>>>>>>>>>> given our current equirectangular format. If there are more >>>>>>>>>>>>>> efficient options on the backend side to achieve the same >>>>>>>>>>>>>> effect, >>>>>>>>>>>>>> we're open to exploring those as well, although frontend >>>>>>>>>>>>>> clients >>>>>>>>>>>>>> would also be phones and tablets, so no heavy operations >>>>>>>>>>>>>> should >>>>>>>>>>>>>> take place on them. >>>>>>>>>>>>>> >>>>>>>>>>>>>> We are currently developing on RK3588 (on Orange Pi 5 >>>>>>>>>>>>>> Plus), >>>>>>>>>>>>>> which by the documentation supports OpenCL 2.2 and Vulkan >>>>>>>>>>>>>> 1.2. It >>>>>>>>>>>>>> has H.264 and H.265 hardware encoders and decoders, which >>>>>>>>>>>>>> we've >>>>>>>>>>>>>> tried using, but v360 filter ended up being the biggest >>>>>>>>>>>>>> bottleneck. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here's one example of how we're currently using this >>>>>>>>>>>>>> filter: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> return [ >>>>>>>>>>>>>>> "ffmpeg",# Decoder settings >>>>>>>>>>>>>>> "-fflags","+genpts+discardcorrupt", >>>>>>>>>>>>>>> "-flags","low_delay", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> "-f","h264", >>>>>>>>>>>>>>> "-i","pipe:0",# input from stdin "-r","30", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # Filter settings "-filter_complex", >>>>>>>>>>>>>>> f"[0:v]scale=1440:720,v360=dfisheye:e:ih_fov=193:iv_fov=193[out]", >>>>>>>>>>>>>>> "-map","[out]", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # Encoder settings "-c:v","libx264", >>>>>>>>>>>>>>> "-preset","ultrafast", >>>>>>>>>>>>>>> "-tune","zerolatency", >>>>>>>>>>>>>>> "-pix_fmt","yuv420p", >>>>>>>>>>>>>>> "-profile:v","main", >>>>>>>>>>>>>>> "-level","3.1", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # Bitrate settings # This ensures that keyframes >>>>>>>>>>>>>>> are pushed >>>>>>>>>>>>>>> frequently enough to keep the # streaming latency low >>>>>>>>>>>>>>> "-g","30",# keyframe interval "-keyint_min","30",# minimum >>>>>>>>>>>>>>> keyframe interval "- sc_threshold","0",# disable scene >>>>>>>>>>>>>>> change >>>>>>>>>>>>>>> detection "- muxpreload","0", >>>>>>>>>>>>>>> "-muxdelay","0", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> "-f","flv",# output format "-rtmp_live","live", >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # e.g.:rtmp://srs-dev/live/livestream >>>>>>>>>>>>>>> output_target, >>>>>>>>>>>>>>> ] >>>>>>>>>>>>>> By my understanding, we likely need a GPU-based v360 filter >>>>>>>>>>>>>> as a >>>>>>>>>>>>>> drop- in replacement for the standard CPU-based v360. I >>>>>>>>>>>>>> think >>>>>>>>>>>>>> that the GPU- based filters would also require `hwupload`/ >>>>>>>>>>>>>> `hwdownload`(?), but I don't have experience with it. >>>>>>>>>>>>>> Personally >>>>>>>>>>>>>> I'm new to FFmpeg, so feel free to correct me if something >>>>>>>>>>>>>> sounds >>>>>>>>>>>>>> off. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The code would ideally be released under a copyleft >>>>>>>>>>>>>> license, I'll >>>>>>>>>>>>>> let Xavier pitch in on this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Does this sound like a proper approach to it and something >>>>>>>>>>>>>> that'd >>>>>>>>>>>>>> be interesting for you to work on? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> whitebox-dev mailing list --dev@ml.whitebox.aero >>>>>>>> To unsubscribe send an email todev-leave@ml.whitebox.aero >>>>>>> -- >>>>>>> Xavier >>>>>>> Whitebox.aero<https://whitebox.aero/> >>>>>> _______________________________________________ >>>>>> whitebox-dev mailing list --dev@ml.whitebox.aero >>>>>> To unsubscribe send an email todev-leave@ml.whitebox.aero
participants (2)
-
milos＠whitebox.aero -
Xavier Antoviaque