[Vive Cosmos OpenXR Hand Tracking Support - Unreal Engine 4.26 Sample Project]

Here is a sample project built using Unreal Engine 4.26 showing how to get started using OpenXR Hand Tracking with Vive Cosmos headsets (Project is attached at the bottom of this page).

Please make sure you follow the instructions specified here first to enable OpenXR in the Vive Console Runtime: 

 

Sample Project

The project comes with 2 pre-installed project plugin:

• Vive Cosmos Controller Plugin defines input subcategories for Cosmos controllers.
• OpenXR Vive Cosmos Controller Plugin allows using Vive Cosmos controllers input for your OpenXR applications as it adds the Vive Cosmos controller interaction profile to OpenXR Input (XR_HTC_vive_cosmos_controller_interaction ) 

We have also enabled the following plugins in the project:

• OpenXR Plugin since we want to build an OpenXR app.
• OpenXR Hand Tracking to support the hand tracking extension of OpenXR

• XR Visualization Plugin allows quickly rendering HMDs,controllers,hand meshes using the relevant data as parameters.This makes it easier to quickly render a representation of a virtual hand based on the information we get about each joint. Of course this is optional and it's not required to use it in your project.

 

 

Implementation

After you open the sample project using Unreal Engine 4.26, please check the Event Graph of the Level Blueprint of the default Level "HandTrackingTest".  

We use the GetMotionControllerData function passing as a parameter the Left or Right Hand and we get back information about the MotionControllerData that can be used to render virtual hands. After that we use the RenderMotionController function from the XRVisualization Plugin to render a virtual representation of hands. You can also break the MotionControllerData structure and use the data about the hands in a different way depending on your use case.

Remember that when asking to "GetMotionControllerData" the C++ side of things will try to get Hand Tracker data via the function "FOpenXRHMD::GetMotionControllerData". While trying to get OpenXRHandTracking data, the engine will get data from the runtime and the .json and .dll files provided as shown below. This is automatically handled after you enable the OpenXR runtime on Vive Console.

 

Here's what you should see after hitting the button Play in VR:

OpenXRHandTest.zip

[Memory leak and crash [unity 2020.1.1f1 / Wave SDK 3.2.0]]

@Linkedoranean The post says that you can't use versions > 2019.3.6, this means you can still use 2019.3.5 for example. Just make sure you use the Unity XR Plugin instead of the Legacy plugin as this will make things easier.
So with Wave XR Plugin we recommend Unity 2019.4 LTS which works 100% or 2020.1 (will work but not stable version yet)

[Working in Unreal Engine with Blueprints]

@ArAnish, If you start from a Blueprint project you can always convert it to a C++ project by creating a new C++ Class from the Unreal Editor (Add New-->C++ Class). Here is a link: https://allarsblog.com/2015/11/05/converting-bp-project-to-cpp/

You can use our SRanipal SDK both in a Blueprint-only project and in a C++ project. We recommend the latter, because then you can easily extend what is available in the plugin, e.g you can create extra BP nodes for your project.

[Extract eye tracker data from Vive Pro Eye]

Hi @Tesi,

we don't provide a scan path at the moment via the SDK but that's something you could implement based on the data provided by the SDK.

[Unreal Eye Tracking Offset]

Ah sorry about that @Tomas_TDFM, @C3D let me re-upload the image here. Let me know if it's visible.

What you need to do is go to SRanipal\Source\SRanipalEye\Private\SRanipalEye_Core.cpp file and replace line 337: 

RayCastDirection = PlayerMainCameraRotation.RotateVector(PlayerMainCameraLocation + CameraGazeDirection * maxDistance);

with the following:

RaycastDirection = (PlayerMainCameraRotation.RotateVector(CameraGazeDirection)*maxDistance) + PlayerMainCameraLocation;

[Unreal Eye Tracking Offset]

Hi @Tomas_TDFM, version 1.3.1.1 should include the fix I mentioned in this thread. I am double-checking that but it looks like the next version should be ready & public early next week. Until then you can always manually change the code if you have a C++ project.

[Getting Started with VRS & Foveated Rendering using HTC Vive Pro Eye & Unreal Engine]

19 hours ago, Stefano said:

@MariosBikos_HTC Thanks for the great tutorial! I'm running into a bit of a problem though. I've followed all the steps for to implement it into 4.24 but it seems to only work when one of my eyes is closed. When I close one of my eyes it tracks the either one just fine, but when I try with both eyes open it doesn't work. I've checked Is Stereo Gaze Data Available and Is Eye Tracker Connected and they are both true. Do you know why this is happening and what I should look into to fix it?

Hi @Stefano,
can you share the VRS settings that you are using when you see this issue? Is this happening with every combination of settings(Foveation Pattern Preset, Foveation Shading Rate)?
Also are you using the latest version of SRanipal SDK? Can you send your logs at @marios_bikos@htc.com so that we can have a look? Anything that can help us reproduce the issue e.g the steps you followed.

[Unreal Eye Tracking Offset]

Hi @C3D,

there was an issue indeed in our SDK and the team managed to fix it. This will be updated in SRanipal v1.3.1.1 but here is an image showing the required code change if you want to fix it earlier than that. Please use this and let us know if it worked.

 

[Unreal Eye Tracking Offset]

Hi @C3D,

I managed to reproduce the issue and have already sent a request for the team to have a look. It looks like the Focus function is not behaving properly but the GetGazeData returns the proper output. Can you check if the GetGazeData works for you until we fix the issue?

[Extract eye tracker data from Vive Pro Eye]

Hi @Tesi,

we provide an Eye Tracking SDK for Vive Pro Eye that is called SRanipal SDK.  It is free for developers to integrate with their project and for the end-users to access via the SRAnipal runtime as the licensing is included as part of Vive Pro Eye hardware platform. Our SDK primarily offers things like feature data (gaze, pupil diameter/position...etc). You can see the full list of features in the image below.

Currently our Vive SRanipal SDK doesn’t allow access to the raw data from the Eye Trackers. Developers can only get access to a specific set of features but not the raw feed from the trackers.

Here is a webinar recording to help you understand more things about Vive Pro Eye and SRAnipal SDK: https://register.gotowebinar.com/recording/212634959163678731

[SRanipal UE4.24 FrameWork Missing and Package Error]

Hey @zgibsontheia,

have you tried initialising the SRanipal runtime first? If you have a Vive Pro Eye device and the SRanipal is properly initialised then ticking the first tick box should initialise eye tracking in Unreal properly.
For the Lip framework you need to have a Lip Tracker device otherwise it won't work so you need to disable it.

Please also try the latest version of sRanipal v1.3.1.0 that comes with some changes on that front: https://developer.vive.com/resources/vive-sense/sdk/vive-eye-tracking-sdk-sranipal/

 

[Eye-Tracking not working: SRanipal requires to update firmware and fails]

Hi all,

SRanipal v1.3.1.0 is now available that should solve the calibration issues mentioned previously in this thread.
Please download it and try again:
https://developer.vive.com/resources/vive-sense/sdk/vive-eye-tracking-sdk-sranipal/

[Wave SDK + Supported Unity versions]

Hi @hieutt,

if you use the Wave Unity XR Plugin  then you can use both Unity 2019.3 and 2019.4 (We recommend 2019.4 as it's an LTS version).
However if you decide to use the Legacy Wave Plugin then you can only use if for versions of Unity up to Unity 2019.3.6, otherwise you will get a memory leak issue. 

We recommend using the XR Plugin as that is future-proof considering the changes in Unity XR Platform. Also the XR Plugin is compatible with newer Unity versions (Unity 2020.1,etc). That's the reason why we focused on that.

[SRanipal UE4.24 FrameWork Missing and Package Error]

Hi @d.b_mann,

you are right, the reason you are getting this error is because after SRanipal v1.3.0.9, Lip Tracking is enabled by default so if you don't want any lip tracking functionality you need to disable the "Enable Lip by Default" settings tick box in SRanipal Project Settings of your UE4 project (see image). If you untick the box you should be able to package without errors. We will update the documentation to highlight this change. Thanks for reporting this 😉

 

 

[Pupil Diameter (Unreal C++)]

Glad that it worked @Franka!
Yes you need to make sure the Eye_Framework is in your scene, as it's required to initialize the Eye tracking process.

@cte the header file in C++ holds the declaration(signature) of a function so it's exactly the same as the function above but without implementation details. You can check the header files to get an idea.

 

[Vive SRWorks SDK Release 0.9.3.0]

Hi @Matyboss, we just released SRworks v0.9.7.1, so you may want to try the new version now that should work with UE 4.24: https://developer.vive.com/resources/vive-sense/sdk/vive-srworks-sdk/0971/

[Meshloader blueprint compile error on UE 4.24.3.]

Hi @ZNMR, we just released SRworks v0.9.7.1, so you may want to try the new version now: https://developer.vive.com/resources/vive-sense/sdk/vive-srworks-sdk/0971/
 

[UE4.25 Compile Error]

Hi @eray_ozr, we just released SRworks v0.9.7.1, can you please try this version with Unreal Engine? https://developer.vive.com/resources/vive-sense/sdk/vive-srworks-sdk/0971/
We recommend using Visual Studio 2017 and not 2019. The new version has been tested with UE 4.24.

[SRanipal_SDK_1.1.0.1 on UE4.25.3]

Hi there!
You can now download SRanipal SDK v1.3.0.9 that has support for UE4.24 and UE 4.25
https://developer.vive.com/resources/vive-sense/sdk/vive-eye-tracking-sdk-sranipal/?site=eu

[Vive Focus Plus UE4 build issue]

Hey @Valvoa, what is your current firmware version on Focus Plus?
I tried UE4.24 and the plugin project that is included in Wave 3.2 SDK and didn't have any issues creating and running a build. Can you please share some logs using adb logcat so that we can see what is going on when the app starts running? Also can you provide the Unreal project logs from {ProjectName}/Saved/Logs?

Did you get any errors during the packaging process of the apk file in Unreal?

[SRanipal_SDK_1.1.0.1 on UE4.25.3]

The plan is for the new version to be released tomorrow (Sept 24th).
Until then, if you want to fix this issue you need to do what I mentioned here:
 

 

[Vive SRWorks SDK Release 0.9.3.0]

Hey @Matyboss there is a going to be a new version of SRworks soon that will add support for UE4.24 that you can use.

[Pupil Diameter (Unreal C++)]

Hey @Franka, you are right, we currently don't have a function exposed to Blueprints for GetPupilDiameter but I have already informed the SDK team about it so that they can add one in a future version of the SDK. Meanwhile, here are some screenshots that will help to add that yourself.

What you need to do is define a GetPupilDiameter function inside USRanipalEye_FunctionLibrary (similarly to how GetPupilPosition is structured) and then also create a function GetPupilDiameter() inside SRanipalEye_Core that will be called from the first function I mentioned. 

Here are the cpp files I used to achieve that. You need to of course modify the header files as well.

 

 

[problem with Blueprint]

Hi @Vita500, there is a going to be a new version of SRWorks available soon (probably by the end of this month) that will support UE4.24 without any issues.
Regarding the problem you mentioned, can you try disabling all the Magic Leap plugins in case there is a conflict?

[SRanipal_SDK_1.1.0.1 on UE4.25.3]