Video performance regression on Harman vs. Adobe Runtime

[xpojosh]

Hello all!

I've seen some significant video performance degradation on Windows since moving to the Harman runtime which appears to be caused by much higher CPU/GPU utilization (for example an Adobe runtime on an Intel i3 system uses 2-4% CPU, 46% GPU when playing a 4K video while the Harman runs at 70-85% CPU and 65% GPU.).

Has anyone else experience this issue?

I've opened up this bug if you'd like to see the particulars:

#2503

Thanks kindly!

josh

[lwmirkk]

Same thing here, for now I'm using AIR 32.0.89 on Windows until one day this resolves.

[ajwfrost]

Without actually checking the metrics (yet) to see whether we can do something about this in the short term, the major change between the Adobe version 32.0.0.89 and subsequent versions (including 32.0.0.116) was that they took out some patent-protected software a/v decoders... so, instead we are using Windows multimedia APIs for decoding the videos. TBH I'm a little surprised that the performance is so significantly worse, I am wondering whether there's something else happening within the Window media foundation integration code that's adding a bottleneck; the earlier decoders were heavily optimized but still just software...

We can look at this but we've also been working (slowly, I'm afraid) on new multimedia integrations that aim to be a bit more optimised and flexible. Currently it's only at the state on Windows where you can play a video with the output being a NativeWindow object (so a bit like being always StageVideo that covers the whole stage area...), the next step is to look at a custom output renderer to be able to push this to VideoTexture, StageVideo, and Video outputs.

But we can have a check to see if there's a quick-win solution to improve things from where they are now...

thanks

[ajwfrost]

Thanks @lwmirkk - if you are able to share any video files that are causing problems, we can take a look in case they're things we can work around or fix in the code... upload link is https://transfer.harman.com/requests/J7GpnrCfcoNGYqn5XOelqc

[DIZ56]

@ajwfrost Will you let me know when the new sdk release that solves the issues we shared and you discovered is available so we may utilize? Thank You in advance.

[ajwfrost]

@DIZ56 just released -> https://airsdk.harman.com/download/51.1.3.4

[lwmirkk]

Thanks @lwmirkk - if you are able to share any video files that are causing problems, we can take a look in case they're things we can work around or fix in the code... upload link is https://transfer.harman.com/requests/J7GpnrCfcoNGYqn5XOelqc

@ajwfrost I sent you the videos. Thanks :)

About the last version: 51.1.3.4 - same problem with this version.
And this version is detected as 51.1.3.3 in Animate, Scout detects as 51.1.3.4.

[ajwfrost]

Thanks, I saw them arrive, we'll take a look

---

And this version is detected as 51.1.3.3 in Animate

Ah yes, because the update of the version that ADT sees is tied up with a script that's run when we build the Android platform, and we didn't update Android this time round (and forgot to trigger that one manually). We'd only tested that the runtime versions have the right value, rather than ADT (which gives the detail to Animate, and which takes the value from the top-level xml file).

thanks

[xpojosh]

Andrew --

Thanks kindly for jumping on this so quickly.

We've done a little more work that may provide some insight here. We ran DXVA checker while running the two different runtimes.

With Adobe 31, we can see that in fact video decoding is being performed by the GPU. With the Harman runtime, the GPU is never called for decoding and thus must be falling back to software (event though an "accelerated" event is reported by the video texture)

See attached DXVA screenshots.

Adobe Traces

Harman Traces

[xpojosh]

@ajwfrost I gather from some comments you have made that the new a/v pipeline development may have stalled out. From the looks of what you have been accomplishing with the Linux runtime, we would be perfectly happy migrating to Linux should hardware-accelerated video decoding via the Videotexture class become available.

Do you have any updates on where video fits into your Linux development roadmap? Is there anything sort of contribution that the community could make to help move this along?

Thanks ever so much.

Cheers!

josh

[ajwfrost]

Hi

The new pipeline isn't progressing very quickly, true .. too many other things that are pushing work on that into the background I'm afraid. We had been looking to do the minimum necessary to get reasonable functionality on some of the problematic platforms i.e. simple playing of video and audio files to normal outputs.

On Linux, we've got support for video via FFMPEG that will be coming shortly, but not hardware accelerated at the moment (some extra steps are required for that). And that is just integrated into the 'normal' video pipeline i.e. via NetStream/Video objects etc, rather than into our new pipeline.

As a more general - hopefully cross-platform - option, we were looking to add in support for NDI decoding and display, and the plan there was to use the new pipeline APIs but done as an ANE, which is where it should then be more feasible for community support to expand upon the capabilities. We just need to then expose the appropriate APIs and demonstrate how it should work. I would hope at this point, we can get hardware-based decoding/display also working. We had started to add ANE APIs to get at the rendering information from within the runtime, with the end goal being to allow ANEs to render to a texture that is then directly used within the runtime on a Stage3D texture object, or as a cached Bitmap display.

When we'd done this kind of thing before - allowing other shared textures to be rendered within AIR - it was very specifically done for Stage3D and a variant Linux using Wayland and OpenGL ES.. it's proving a little more tricky to make this work with all the various platforms and rendering mechanisms that we have! So I am also wondering whether we should start off by only implementing it on one mechanism - most likely, via OpenGL ES. We're getting OpenGL ES support on Windows shortly, via ANGLE, and the same should then be possible for macOS and Linux (as an optional feature i.e. you have to choose to render via ANGLE) - which then gives us a lot more of a consistent codebase/feature-set to integrate these kinds of additional capabilities.

Hope that's useful..

[BwackNinja]

It's fantastic that this is moving forward, but I believe that StackOverflow link is rather old.

Here's a complete and documented example using VA-API decoding [1]. VA-API is basically the only hardware acceleration API that needs to be supported for desktop Linux[2][3]. This example supports EGL and GLX rendering with no roundtrips to the CPU (EGL needs to be used for Wayland support).

If using a hardware acceleration API other than VA-API, the only lines strictly dependent on VA-API otherwise are 568-621, which are using the converting the frame to be used by OpenGL.

[1] https://gist.github.com/kajott/d1b29c613be30893c855621edd1f212e

[2] NVIDIA support isn't a worry since there exists https://github.com/elFarto/nvidia-vaapi-driver which is widely known, and https://cgit.freedesktop.org/vaapi/vdpau-driver for much older hardware. Neither Firefox nor Chrom{e,ium} support anything other than VA-API.

[3] Less common on desktop Linux, but relevant because of aarch64 support, https://github.com/bootlin/libva-v4l2-request exists, wrapping the V4L2 Request API which is a more common way of implementing hardware accelerated decoding/encoding on Linux ARM devices.

[ajwfrost]

@BwackNinja thank you very much! That's good to know (and yes, having too much obsolete documentation is sometimes a challenge) - the example code there looks pretty straightforward, plus using EGLImage this is something we should be able to use certainly in Stage3D and hopefully for StageVideo too.

Let me see if we can move this forward sooner rather than later!

[BwackNinja]

@ajwfrost Has there been any progress on Linux hardware accelerated video decoding? It would be a winter holiday miracle to have a beta version to test! If not, do you have any idea when we might hope to see it?

[xpojosh]

I agree with @BwackNinja here: progress in this area would make all my holiday dreams come true!

#makeOurDreamsComeTrue
#doItForTheKids

[ajwfrost]

Hi @xpojosh @BwackNinja - so we had been working on the underlying video decoding for Linux via FFMPEG and had to also do the audio decoding this way, which is pretty much complete - but that is still using the internal mechanisms for the 'Video' object i.e. we pull video into the CPU to do the rendering.I'm hoping to release that (in 51.2) as soon as possible in the new year; we've got a 51.1.3 bug fix release coming out this side of Christmas but not a major update...

The other activity was to extend the ANE interface slightly so that we can use the above example code and run that snippet in a separate thread in an ANE, and render it to a Texture object in Stage3D. This would give us the whole thing fully hardware accelerated and should then work on Linux; there's no reason why that ANE couldn't then be extended to other platforms. All of this should be enabled in 51.2, which has become a lot bigger than we originally planned i.e. a lot going into it now..

I'd hoped we could try to get a beta build out for this, but with only a week before Christmas, this may be too much of a challenge, sorry!

[DIZ56]

Andrew,Thank you for your reply.  Fyi: It never did it with 29 or 32 version we used however appreciate you found issue with it. The thread seems to have disappeared or atleast on my end. Do you have a ETA for fix?  Best Regards,Keith Disbro317 800 ***@***.*** Jan 6, 2025 1:33 PM, Andrew Frost ***@***.***> wrote: Thanks @DIZ56 @istoan - that usage showed we had a thread synchronisation issue because two different threads were both trying to stop the feeder thread due to the seek happening right at the start; we also then found a recursive issue with the metadata object being dispatched repeatedly because of the seek(0) call - so we've had to ensure we don't trigger onMetaData when we're in the middle of a seek. But we should be able to fix this use case in our next release. I'm not actually sure how come this worked with the Adobe version of the SDK because the dispatch of metadata when seeking to zero appears to have been present then too. Actually I've just run this on AIR32 and I do get the same: Starting NS1: NetConnection.Connect.Success NS1: NetStream.Play.Start NS1: NetStream.Buffer.Full metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify NS1: NetStream.Buffer.Full metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify NS1: NetStream.Buffer.Full metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify NS1: NetStream.Buffer.Full metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify NS1: NetStream.Buffer.Full metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.SeekStart.Notify NS1: NetStream.Seek.Notify NS1: NetStream.Buffer.Full metadata: duration=46.439909297052154 width=1280 height=720 framerate=undefined NS1: NetStream.Seek.Failed and then it plays. So, sounds like the main issue in our build was the thread synchronisation; once that's sorted, behaviour is the same; but we can make it a little better by only dispatching the metadata when the stream actually starts up for the first time rather than after every seek(0). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>