VIVE Hand Tracking Camera Resolution

[Xyah]

Hello, We have been testing around the VIVE Hand Tracking API + Unreal plugin with our VIVE Cosmos and after some profiling found out that as soon as detection started, FPS dropped from 90 FPS to 45~60 FPS. The following is the information of the test: - Test Computer: Intel i7 7700k, Nvidia GTX 1060 - Unreal 4.24: Blank map with nothing but a SkySphere, Hand Tracking Provider Blueprint Actor and the Cartoon Hands Blueprint Actor (both actors provided by the VIVE Hand Tracking Unreal plugin). - Performance: In this blank level, the FPS still dropped then to 45~60FPS, and this is not an acceptable frame rate for an optimal VR experience. - What we found out: We know that the resolution of the Camera affects largely on the performance of the image processing and that's why the VIVE VR (lower resolution camera) outperforms the VIVE Cosmos (higher resolution camera) in terms of hand tracking performance (and overall rendering of the scene). What we found out is that if the overall resolution is decreased to lowest as possible, we would get back at 90 FPS (although it might only be due to the performance gains in overall scene rendering). - What we want to ask you: Is there a way to change the Camera Resolution of the VIVE Cosmos? We want to make image processing faster to process and take less GPU resources. It does not matter if it's via using another SDK, we just need to know a way to do so.

[zzy]

Hi @Xyah

Are you using Cosmos XR or original Cosmos? Camera resolution of original Cosmos is similar to Vive Pro, so I think it's not the root cause.

Here are some questions/suggestions for the general frame drop problem:

1. How about running the pre-built sample (built by Unity) and see if there is frame drop? You can use SteamVR to check if there is frame drop.
   1. I would like to know if this is specific to Unreal or not.
   2. We do have a test laptop with 1060, but Unity sample seems to be working fine with Vive Pro, Cosmos and Cosmos XR.
2. Does it mean that Vive (or Vive Pro?) works fine on your PC without frame drop?
3. Can you check if the frame drop is CPU bounded or GPU bounded? We also uses some CPU resources in our pipeline, and if CPU is 100%, this can easily cause lagging.
   1. From what I know, if Vive Console is running for a long time, it may use up to 100% CPU and slows down the PC. You can restart Vive Console in such case.

[Xyah]

Hi @zzy!

Thank you for your reply!

We tested on the original Cosmos. We have the VIVE, VIVE Pro and VIVE Cosmos. We tested the Hand Tracking for both (original) VIVE and VIVE Cosmos. We did not test on VIVE Pro since we know that the camera resolution is the same and the performance of the Hand Tracking is the same as per the given table:

 

VIVE worked fine (no FPS drops and maintained a steady 90 FPS in the 1060 i7 7700k PC). We used Steam VR and there is definitely frame drop spikes when Hand Tracking is taking place. To give you more detailed information here is the order of how we saw things:

1) Game Started (and Start Detection immediately called). No Frame Drops (since StartDetection is apparently non-blocking). stable 90 FPS.

2) When Detection is in Running State, the FPS drops to 45~60 FPS. 

3) When Detection is also running and the hands are completely out of the VIVE Cosmos sight, the FPS sometimes goes up to 90 FPS again.

While all this was happening, we saw Task Manager and Performance Profiling. Task Manager showed no indication of a clear bottleneck (both CPU and GPU were well under 60%). Performance Profiling in Unreal showed a low tick rate (i.e. around 45~60 FPS) but no clear indication of something within Unreal hindering it. 

 

We want to test whether Camera Resolution affects this performance (as per the table above). Is changing camera resolution possible?

 

 

 

[zzy]

Hi @Xyah

Thanks for the test. As far as I know, there is no way to change camera resolution of original Cosmos.

From your results, it seems the FPS drop only happens when hand is visible. We will check if we can reproduce it on our side. Meanwhile, can you confirm if it's same behavior in Unreal and Unity (pre-built sample)?

[Xyah]

Thank you for the quick response @zzy! 
We will test both Unreal and Unity once again!

[Xyah]

Hello @zzy!

I currently do not have access to the test computer ( gtx 1060). I will be able to give you test results with a 1660 but I do not know how big of a difference that will make.

I have a few questions I would like to ask you!

1) Where can I get the pre-built sample of Unity? (since we do not have Unity installed in our PCs)

2) Is there consideration in making the Camera Input Resolution adjustable so as to change the overall performance of the Hand Tracking SDK? If so, is there an ETA on this?

3) Is the next update of the Hand Tracking SDK addressing  performance issues or is just focused towards accuracy?

Thank you so much! 

[zzy]

Hi @Xyah

I have tested using our test machines (1060 & 2060) today. I'll answer your question first and then update my findings.

1. You can find the prebuilt samples in the zip you downloaded from here: https://developer.vive.com/resources/knowledgebase/vive-hand-tracking-sdk/. "Prebuilt samples" folder contains binaries built using Unity.
2. I think this requires camera API to support input resolution settings, but currently no API has such support (SteamVR or Vive Console), so we probably can do nothing about this.
3. We did some performance tuning, but I think that not this big to solve your problem.

About my findings, I think this might be a problem from Unreal side instead of hand tracking sdk. Here is my experiment, I use SteamVR frame timer to check the FPS and GPU times. In all my experiments, CPU is the bottleneck.

1. I run Unity sample on 1060 & 2060, GPU time is larger when hands are visible, but the frame time is within 11ms (90 FPS).
2. I run Unreal app, GPU time is significantly larger than Unity version, about 2x times are used in GPU.
3. I tried to create a new default level, without hand related stuffs. It consumes very large GPU times, even worse than Unity sample with hand. This makes me think the problem is on Unreal side.

Please let me know if this is the same situation on your side.

[Xyah]

Hi @zzy

I managed to test with the 1660 both Unity Samples and a built version of a test project using the Hand Tracking plugin for Unreal.

Unity's performance was of about 10 ms and Unreal's was about 16~20 ms (when Deferred Rendering used)

 Unreal might be having a slower performance because it uses the default Deferred Rendering rather than forward rendering. Even after setting Forward Rendering, through profiling we saw that deferred rendering was still being used. We managed to have a case where Forward Rendering was used (no performance hit from Deferred Rendering) and the performance result was similar to that of Unity (within 11 ms).

However we noticed that the Hand Tracking SDK in both Unity and Unreal offer a frame time of just below 11 ms, which is not desirable when we are talking about only having very basic objects in the scene. It seems that Hand Tracking SDK is not in a current state to be implemented in games.

Also I gathered more information about the "off screen" performance gain. I disabled the "Hide in Game" functionality when the Hands are not detected so as to always show the Hand Meshes .  The performance drop seems to be happening when the API is detecting the Hand. Once it stops detecting the performance increases even if the Hand Mesh is still visible, so that sudden shift in performance seems to be happening within the Hand Tracking API.

Can I ask, is there anyway we can get a more "low" level API for this? It's very frustrating only being offered 4 functions without room to configure more and play around with performance. I would appreciate it!

[zzy]

Hi @Xyah

Thanks for the test. Since the deep learning model are quite computational heavy, it's natural that it might eat up a lot of GPU resources.

If you want to use lighter version of Hand Tracking, you can change the mode of hand tracking to 3D Point, which should be much faster than skeleton mode, although returns limited result.

We plan to release new version before end of this month, which should bring some performance improvement. You can have a test when the new version is released. We will also keep improving performance, as there are several items in our backlog, but considering current SDK status, accuracy might take higher priority.

Here are some references:

Different between different modes: https://hub.vive.com/storage/tracking/overview/feature.html#hand-positions-modes

Information about how to change mode in Unreal: https://hub.vive.com/storage/tracking/unreal/usage.html#start-stop-detection

[zzy]

Hi @Xyah

v0.9.3 is now released, you can download it from here. It improves GPU usage in hand tracking sdk on Windows. Please have a try.