Add SVC and Simulcast support for AV1 and VP9. #2350

jallamsetty1 · 2023-09-13T16:19:31Z

Both Simulcast and SVC modes are supported for AV1 and VP9. The default mode for both of them is SVC and is enabled by default unless it is explicitly disabled from config.js.

videoQuality: {
      av1: {
            maxBitratesVideo: {
               low: 100000,
               standard: 300000,
               high: 1000000,
               ssHigh: 2500000
            },
            scalabilityModeEnabled: true,
            useSimulcast: false,
            useKSVC: true
        },
        h264: {
            maxBitratesVideo: {
                low: 200000,
                standard: 500000,
                high: 1500000,
                ssHigh: 2500000
            },
            scalabilityModeEnabled: true
        },
        vp8: {
            maxBitratesVideo: {
                low: 200000,
                standard: 500000,
                high: 1500000,
                ssHigh: 2500000
            },
            scalabilityModeEnabled: false
        },
        vp9: {
            maxBitratesVideo: {
                low: 100000,
                standard: 300000,
                high: 1200000,
                ssHigh: 2500000
            },
            scalabilityModeEnabled: true,
            useSimulcast: false,
            useKSVC: true
        }
}

jallamsetty1 · 2023-09-13T20:25:56Z

Depends on jitsi/jitsi-videobridge#1998

saghul

Left some comments!

modules/RTC/TPCUtils.js

service/RTC/VideoEncoderScalabilityMode.ts

saghul · 2023-09-14T09:59:15Z

modules/RTC/TPCUtils.js

+        if (videoQualitySettings) {
+            for (const codec of VIDEO_CODECS) {
+                const bitrateSettings = videoQualitySettings[codec]?.maxBitratesVideo
+                    ?? videoQualitySettings[codec.toUpperCase()]?.maxBitratesVideo


Why don't we have the codecs in upper case in the constant to avoid having to uppercase here?

I am trying to make sure that the code is case insensitive for the codec value in config.js. If we have them in upper case in the constant, then we need to check if the codec value is in lower case in config.js.

But in the same line you are indexing it both ways. It looks super weird.

modules/RTC/TPCUtils.js

modules/RTC/TraceablePeerConnection.js

modules/browser/BrowserCapabilities.js

service/RTC/StandardVideoSettings.ts

saghul

Left some more comments, PTAL!

modules/RTC/TPCUtils.js

saghul · 2023-09-21T12:14:58Z

modules/RTC/TPCUtils.js

+        if (videoQualitySettings) {
+            for (const codec of VIDEO_CODECS) {
+                const bitrateSettings = videoQualitySettings[codec]?.maxBitratesVideo
+                    ?? videoQualitySettings[codec.toUpperCase()]?.maxBitratesVideo


But in the same line you are indexing it both ways. It looks super weird.

modules/RTC/TPCUtils.js

modules/RTC/TPCUtils.spec.js

modules/RTC/TraceablePeerConnection.js

service/RTC/StandardVideoSettings.ts

Multi-encoding simulcast for H.264 is supported now because of the AV1 DD support added on the bridge side.

Expect the videoQuality settings in the new format, deprecated configs are still supported. videoQuality: { AV1: { maxBitratesVideo: { low: 100000, standard: 300000, high: 1000000, ssHigh: 1500000 }, useScalabilityModeAPI: { enabled: true, useSimulcast: true, //(defaults to SVC), useL3T3: true //(defaults to L3T3_KEY) }, }, H264: { maxBitratesVideo: { low: 200000, standard: 500000, high: 1500000, ssHigh: 2500000 } }, VP8: { maxBitratesVideo: { low: 200000, standard: 500000, high: 1500000, ssHigh: 2500000 } }, VP9: { maxBitratesVideo: { low: 100000, standard: 300000, high: 1200000, ssHigh: 2500000 }, useScalabilityModeAPI: { enabled: true, useSimulcast: true, //(defaults to SVC), useL3T3: true //(defaults to L3T3_KEY) }, } }

Both Simulcast and SVC modes are sipported for AV1 and VP9. The default mode for both of them is SVC and is enabled by default unless it is explicitly disabled from config.js. videoQuality: { av1: { maxBitratesVideo: { low: 100000, standard: 300000, high: 1000000, ssHigh: 2500000 }, scalabilityModeEnabled: true, useSimulcast: false, useL3T3: true }, h264: { maxBitratesVideo: { low: 200000, standard: 500000, high: 1500000, ssHigh: 2500000 }, scalabilityModeEnabled: true }, vp8: { maxBitratesVideo: { low: 200000, standard: 500000, high: 1500000, ssHigh: 2500000 }, scalabilityModeEnabled: false }, vp9: { maxBitratesVideo: { low: 100000, standard: 300000, high: 1200000, ssHigh: 2500000 }, scalabilityModeEnabled: true, useSimulcast: false, useL3T3: true } }

Add more unit tests.

Also read the deprecated max bitrates correctly, add unit tests to test it.

It makes more sense to call it spatial scalability than simulcast now that full SVC support is available.

modules/RTC/TPCUtils.js

saghul · 2023-10-31T15:19:34Z

modules/RTC/TPCUtils.js

+     */
+    calculateEncodingsScaleFactor(localVideoTrack, codec, maxHeight) {
+        if (this.pc.isSpatialScalabilityOn() && this.isRunningInSimulcastMode(codec)) {
+            return;


Why not return 1?

The scale factor needs to be reconfigured (for layer suspension) only for the full SVC case. For simulcast case, they remain constant.

saghul · 2023-10-31T15:20:11Z

modules/RTC/TPCUtils.js

+                return height / activeEncoding.scaleResolutionDownBy;
+            }
+
+            return null;


Why not make this 0?

Actually, this method returns null for cases when the send resolution cannot be determined. I think it makes sense to return null here.

saghul · 2023-10-31T15:21:42Z

modules/RTC/TraceablePeerConnection.js

+     * AV1 Dependency Descriptor header extensions is offered by Jicofo. H.264 simulcast is also possible when these
+     * header extensions are negotiated.
+     */
+    this._supportsAv1HeaderExts = false;


Wouldn't renaming this to supportsDDHeaderExt make more sense?

saghul · 2023-10-31T15:22:39Z

modules/RTC/TraceablePeerConnection.js

+
+    const parsedSdp = transform.parse(description.sdp);
+    const mLines = parsedSdp.media.filter(m => m.type === MediaType.VIDEO);
+    const extUri = 'https://aomediacodec.github.io/av1-rtp-spec/#dependency-descriptor-rtp-header-extension';


Maybe extract this to the top since it's a module level constant.

saghul · 2023-10-31T15:23:08Z

modules/RTC/TraceablePeerConnection.js

+
+            if (this._supportsAv1HeaderExts && shouldNegotiateHeaderExts && headerIndex < 0) {
+                mLine.ext.push({
+                    value: 11,


11 is the header ext id used by Jicofo/bridge for the DD RTP header extension. I have declared it as a variable now.

saghul · 2023-10-31T15:25:02Z

modules/xmpp/JingleSessionPC.js

@@ -403,23 +401,25 @@ export default class JingleSessionPC extends JingleSession {
        pcOptions.capScreenshareBitrate = false;
        pcOptions.codecSettings = options.codecSettings;
        pcOptions.enableInsertableStreams = options.enableInsertableStreams;
-        pcOptions.videoQuality = options.videoQuality;
+
+        if (options.videoQuality) {


What does this do?

This converts all the videoQuality key values to lowercase so that the code is case insensitive.

saghul

Amazing work! Left some nmon-blocking comments, which can be handled in subsequent PRs since I suppose we might need to tweak things a bit.

Also, is mobile ok with this change?

jallamsetty1 · 2023-10-31T16:30:19Z

Amazing work! Left some nmon-blocking comments, which can be handled in subsequent PRs since I suppose we might need to tweak things a bit.

Also, is mobile ok with this change?

Actually, thats a good idea. I will merge this and open a follow-up PR. I don't see many changes though. Yes, no issues with mobile.

saghul · 2023-10-31T16:40:00Z

Amazing work! Left some nmon-blocking comments, which can be handled in subsequent PRs since I suppose we might need to tweak things a bit.

Also, is mobile ok with this change?

Actually, thats a good idea. I will merge this and open a follow-up PR. I don't see many changes though. Yes, no issues with mobile.

Let's go then! 🙌

Add SVC and Simulcast support for AV1 and VP9. The default mode is the full SVC mode for VP9. It works as expected with the current version of JVB. AV1 in the client is only supported when the bridge negotiates the Dependency Descriptor extension headers which are needed for reading the layer information for AV1 and H.264 video frames. * feat: Enable H.264 simulcast support. Multi-encoding simulcast for H.264 is supported now because of the DD header ext support added on the bridge side. * fix(video-quality): Add default bitrates for all codecs. Expect the videoQuality settings in the new format, deprecated configs are still supported. videoQuality: { AV1: { maxBitratesVideo: { low: 100000, standard: 300000, high: 1000000, ssHigh: 1500000 }, useScalabilityModeAPI: { enabled: true, useSimulcast: true, //(defaults to SVC), useKSVC: true //(defaults to L3T3_KEY) }, }, H264: { maxBitratesVideo: { low: 200000, standard: 500000, high: 1500000, ssHigh: 2500000 } }, VP8: { maxBitratesVideo: { low: 200000, standard: 500000, high: 1500000, ssHigh: 2500000 } }, VP9: { maxBitratesVideo: { low: 100000, standard: 300000, high: 1200000, ssHigh: 2500000 }, useScalabilityModeAPI: { enabled: true, useSimulcast: true, //(defaults to SVC), useKSVC: true //(defaults to L3T3_KEY) }, } } * Disable encodings when requested height for screenshare is 0. * feat: add unit tests for scalability modes. * ref: Move all encoding configuration calculation to TPCUtils. * Add more unit tests for VP9 K-SVC mode. * Make codec name in settings case insensitive. Also read the deprecated max bitrates correctly, add unit tests to test it. * ref(TPC): isSimulcastOn -> isSpatialScalabilityOn. It makes more sense to call it spatial scalability than simulcast now that full SVC support is available. * fix: Negotiate AV1 DD header exts only for AV1 and H.264. * fix: Rename setting useL3T3Mode->useKSVC and adjust the defaults.

jallamsetty1 changed the title ~~Feat/add av1 support~~ Add SVC and Simulcast support for AV1 and VP9. Sep 13, 2023

jallamsetty1 force-pushed the feat/add-AV1-support branch from b65fc87 to a4d8fed Compare September 13, 2023 19:21

saghul reviewed Sep 14, 2023

View reviewed changes

jallamsetty1 force-pushed the feat/add-AV1-support branch 2 times, most recently from aae4fb0 to 36cfbcd Compare September 20, 2023 15:21

jallamsetty1 requested a review from saghul September 20, 2023 16:18

saghul reviewed Sep 21, 2023

View reviewed changes

jallamsetty1 added 8 commits September 26, 2023 19:47

feat: Enable H.264 simulcast support.

fd107c9

Multi-encoding simulcast for H.264 is supported now because of the AV1 DD support added on the bridge side.

Address review comments.

af81c5e

Disable encodings when requested height for screenshare is 0.

9d67012

feat: add unit tests for scalability modes.

f654bb0

fix: Check if the codec values are in uppercase in the settings.

8c7b3f9

ref: Move all encoding configuration calculation to TPCUtils.

eea4b9e

Add more unit tests.

jallamsetty1 force-pushed the feat/add-AV1-support branch from 72b525a to eea4b9e Compare September 26, 2023 23:58

fix failing unit test after rebase

24ca340

jallamsetty1 force-pushed the feat/add-AV1-support branch from 516b643 to 24ca340 Compare September 27, 2023 13:09

jallamsetty1 added 4 commits October 6, 2023 15:37

Add more unit tests for VP9 K-SVC mode.

fb150d4

Make codec name in settings case insensitive.

ad6e1af

Also read the deprecated max bitrates correctly, add unit tests to test it.

ref(TPC): isSimulcastOn -> isSpatialScalabilityOn.

72e0736

It makes more sense to call it spatial scalability than simulcast now that full SVC support is available.

fix: Negotiate AV1 DD header exts only for AV1 and H.264.

c95d187

jallamsetty1 force-pushed the feat/add-AV1-support branch from 2303526 to c95d187 Compare October 17, 2023 19:54

fix: Rename setting useL3T3Mode->useKSVC and adjust the defaults.

c4afa74

jallamsetty1 requested a review from saghul October 27, 2023 18:33

saghul reviewed Oct 31, 2023

View reviewed changes

modules/RTC/TPCUtils.js Show resolved Hide resolved

saghul reviewed Oct 31, 2023

View reviewed changes

saghul approved these changes Oct 31, 2023

View reviewed changes

jallamsetty1 merged commit 7b6ce94 into jitsi:master Oct 31, 2023
1 check passed

jallamsetty1 deleted the feat/add-AV1-support branch October 31, 2023 17:40

danryu mentioned this pull request Jan 8, 2024

AV1 support? jitsi/jitsi-videobridge#1977

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SVC and Simulcast support for AV1 and VP9. #2350

Add SVC and Simulcast support for AV1 and VP9. #2350

jallamsetty1 commented Sep 13, 2023 •

edited

Loading

jallamsetty1 commented Sep 13, 2023

saghul left a comment

saghul Sep 14, 2023

jallamsetty1 Sep 14, 2023

saghul Sep 21, 2023

saghul left a comment

saghul Sep 21, 2023

saghul Oct 31, 2023

jallamsetty1 Oct 31, 2023

saghul Oct 31, 2023

jallamsetty1 Nov 1, 2023

saghul Oct 31, 2023

saghul Oct 31, 2023

saghul Oct 31, 2023

jallamsetty1 Nov 1, 2023

saghul Oct 31, 2023

jallamsetty1 Nov 1, 2023

saghul left a comment

jallamsetty1 commented Oct 31, 2023

saghul commented Oct 31, 2023

Add SVC and Simulcast support for AV1 and VP9. #2350

Add SVC and Simulcast support for AV1 and VP9. #2350

Conversation

jallamsetty1 commented Sep 13, 2023 • edited Loading

jallamsetty1 commented Sep 13, 2023

saghul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saghul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saghul left a comment

Choose a reason for hiding this comment

jallamsetty1 commented Oct 31, 2023

saghul commented Oct 31, 2023

jallamsetty1 commented Sep 13, 2023 •

edited

Loading