3D pinch interaction and grab object interactions do not work

[cpinter]

• Build the latest Slicer 5.3.0 with the latest SlicerVR
• Copy the manifest json files into the folder of Slicer.exe
• Start VR
• I used an HP Reverb 2 headset

None of the interactions worked, including flying, grab&move objects, and 3D pinch. The latter two at least worked a few months ago. Maybe some of the latest commits in VTK or here broke them?

I updated OpenVR to the latest version and tried the same Slicer 5.3.0 + SlicerVR on another computer with HTC Vive Pro but there was no change.

[cpinter]

This is due to the RecognizeComplexGesture function being commented out since 1c36e40

[cpinter]

The problem is that getting the controller physical pose matrices return identity matrix (here). Otherwise I think it should work so we only need to find how to get those poses. @LucasGandel do you have any off-hand idea by any chance? Maybe after the API change it is retrieved another way? Thanks!

[jcfr]

VTK / MR-9892 /Re-introduce support for custom logic handling VR complex gesture

Related to RecognizeComplexGesture:

• MR-9892 | ENH: Re-introduce support for custom logic handling VR complex gesture will be integrated into upstream VTK.

• In the meantime, worth noting that:

  • All commits from MR-9892 have been added to the updated VTK being integrated through:
    • ENH: Update VTK from 9.1.20220125 to 9.2.20230607 Slicer/Slicer#7010
  • The first commit of MR-9892 titled Re-introduce support for custom logic handling VR complex gesture is already in the current version of Slicer/[email protected] as Slicer/VTK@6efd97c

Origin of the "regression"

None of the interactions worked, including flying, grab&move objects, and 3D pinch

Indeed, see:

• Address issues related to integration with VTK 9.1 #91

How to get the VR poses ?

how to get those poses

In the context of OpenVR, corresponding API has been re-introduced in Slicer/VTK. Relevant commit titles are:

Title	slicer-v9.1.20220125-efbe2afc2	slicer-v9.2.20230607-1ff325c54
ENH: Re-introduce OpenVR API for retrieving last OpenVR pose	Slicer/VTK@c333750	Slicer/VTK@c6189a4
BUG: Ensure a valid OpenVR pose is returned	Slicer/VTK@b74591f	Slicer/VTK@0227875

Since this approach is specific to RenderingOpenVR, I suggest we work with @LucasGandel to come with an approach that would also work with RenderingOpenXR.

[cpinter]

Thanks @jcfr !

I updated the hash in an attempt to use the new SlicerVTK branch, and got

these error messages when building

    vtkMeshQuality.cxx

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1290,10): error C2653: 'verdict': is not a class or namespace name [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1290,19): error C3861: 'tri_area': identifier not found [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1303,10): error C2653: 'verdict': is not a class or namespace name [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1303,19): error C3861: 'tri_edge_ratio': identifier not found [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1316,10): error C2653: 'verdict': is not a class or namespace name [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1316,19): error C3861: 'tri_aspect_ratio': identifier not found [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1329,10): error C2653: 'verdict': is not a class or namespace name [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1329,19): error C3861: 'tri_radius_ratio': identifier not found [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1342,10): error C2653: 'verdict': is not a class or namespace name [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1342,19): error C3861: 'tri_aspect_frobenius': identifier not found [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1355,10): error C2653: 'verdict': is not a class or namespace name [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]

C:\d\S5R\VTK\Filters\Verdict\vtkMeshQuality.cxx(1355,19): error C3861: 'tri_minimum_angle': identifier not found [C:\d\S5R\VTK-build\Filters\Verdict\FiltersVerdict-objects.vcxproj] [C:\d\S5R\VTK.vcxproj]
...

(and a lot of the same)

For now I'll just try to fix the relevant part in vtkOpenVRRenderWindow.cxx that always sets the pose to null, but I thought this would be good to know.

[LucasGandel]

@cpinter The verdict related issue can be fixed by removing the ThirdParty/verdict folder in your build directory and rebuilding.

Besides that, I confirm we must work on an approach that is common to both OpenXR and OpenVR and which does not expose specific OpenVR/OpenXR types in the public API.

[jcfr]

re: Slicer + latest VTK
There are errors on the Slicer side that I am fixing. So far, since these are located in CLIs, you should be able to comment the code.

[cpinter]

I just downloaded and tried the latest nightly Slicer 5.3.0-2023-06-12, and 3D pinch works for me.

Note that I had to first work around the json manifest issue (see #115 (comment))

After that, this is how I could use the interactions. I use an HP Reverb 2 headset.

2023-06-13.16-57-38.mp4

[jcfr]

After that, this is how I could use the interactions. I use an HP Reverb 2 headset.

Fore reference this was fixed by 49f1896

Thanks @cpinter for working on the fix.

@LucasGandel The current approach makes use of the vtkOpenVRRenderWindow::GetOpenVRPose method specific to Slicer/VTK¹, I am wondering if we could generalize and have an API common to OpenVR and OpenXR.

Footnotes

1. https://github.com/Slicer/VTK/compare/97a187572d4000cd820f9fc887f21eaf0bde857c...b74591f27e11bb7d8f750fd96ae85fa48cf67e63 ↩

[cpinter]

Since the interactions themselves are fixed I think we can close this.