ARFoundation sample issue

[RRob]

Hello,

I'm having an issue with the "Assets\Samples\VIVE Hand Tracking SDK\1.0.0\Sample\ARFoundation.unity" scene.
For some reason the camera passthrough does not work.

My setup is:
Unity: 2021.1.21f1
Packages:



Description:
When compiling and running it on my phone, the screen is black but the hand tracking is working.
I have narrowed it down to the GestureProvider.cs script. If it is disabled, camera passthrough works but it defeats the purpose.

Using logcat I managed to extract this:

2021/09/19 22:31:56.287 5246 5289 Error native ################ ARCore Native Error ##################
2021/09/19 22:31:56.287 5246 5289 Error native BUILD_CHANGELIST:387218412
2021/09/19 22:31:56.287 5246 5289 Error native BUILD_BASELINE_CHANGELIST:385683590
2021/09/19 22:31:56.287 5246 5289 Error native ################### Stack Trace Begin ################
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/infrastructure/cameras_external_state.cc:193	https://cs.corp.google.com/piper///depot/google3/third_party/arcore/ar/infrastructure/cameras_external_state.cc?g=0&l=193
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/infrastructure/cameras_async.cc:162	https://cs.corp.google.com/piper///depot/google3/third_party/arcore/ar/infrastructure/cameras_async.cc?g=0&l=162
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/infrastructure/android_data_source.cc:635	https://cs.corp.google.com/piper///depot/google3/third_party/arcore/ar/infrastructure/android_data_source.cc?g=0&l=635
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/core/session.cc:1762	https://cs.corp.google.com/piper///depot/google3/third_party/arcore/ar/core/session.cc?g=0&l=1762
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/core/c_api/session_lite_c_api.cc:76	https://cs.corp.google.com/piper///depot/google3/third_party/arcore/ar/core/c_api/session_lite_c_api.cc?g=0&l=76
2021/09/19 22:31:56.287 5246 5289 Error native ################### Stack Trace End #################
2021/09/19 22:31:56.287 5246 5289 Error native 
2021/09/19 22:31:56.287 5246 5289 Error native ################### Undecorated Trace Begin  #################
2021/09/19 22:31:56.287 5246 5289 Error native FAILED_PRECONDITION: 
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/core/session.cc:1762
2021/09/19 22:31:56.287 5246 5289 Error native 
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/infrastructure/android_data_source.cc:635
2021/09/19 22:31:56.287 5246 5289 Error native 
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/infrastructure/cameras_async.cc:162
2021/09/19 22:31:56.287 5246 5289 Error native 
2021/09/19 22:31:56.287 5246 5289 Error native ARCoreError: third_party/arcore/ar/infrastructure/cameras_external_state.cc:193
2021/09/19 22:31:56.287 5246 5289 Error native Operation required Started but state is Error
2021/09/19 22:31:56.287 5246 5289 Error native === Source Location Trace: ===
2021/09/19 22:31:56.287 5246 5289 Error native third_party/arcore/ar/infrastructure/cameras_external_state.cc:327
2021/09/19 22:31:56.287 5246 5289 Error native third_party/arcore/ar/infrastructure/cameras_external_state.cc:193
2021/09/19 22:31:56.287 5246 5289 Error native third_party/arcore/ar/infrastructure/cameras_async.cc:162
2021/09/19 22:31:56.287 5246 5289 Error native third_party/arcore/ar/infrastructure/android_data_source.cc:635
2021/09/19 22:31:56.287 5246 5289 Error native third_party/arcore/ar/core/session.cc:1762
2021/09/19 22:31:56.287 5246 5289 Error native 
2021/09/19 22:31:56.287 5246 5289 Error native ################### Undecorated Trace End  #################
2021/09/19 22:31:56.287 5246 5289 Error native 
2021/09/19 22:31:56.287 5246 5289 Error native session_manager.cc:115 ArPresto::ArSession_update failed with status -2.

This seems not to happen when I disable the GestureProvider.
I could not find the sources for the SDK so I could not dig further.
If anyone has any idea, it would help a lot.

Photo with GestureProvider enabled:


Photo with GestureProvider disabled:


Thanks.

[zzy]

Hi @RRob

Can you please share the full adb log? I would suspect that hand tracking plugin doesn't detected arcore and thus getting camera stream using ndk camera, not from arcore apis.

Although you may have done this, please make sure you follow the instructions in the ARFoundation sample to replace "AR Session" and "AR Session Origin" game object with latest version.

Can you please also try Unity LTS versions? I have previously tested on 2019.4 and 2020.3 which should work fine. I'll check if I can reproduce this on my side.

[RRob]

Hey @zzy

I've tried with 2020.3.11f1 as I already had it installed and had the same result.
My phone is a Xiaomi P20 Pro in case that matters and also tried with a Samsung A50, both of which are on the ARCore supported list.
I've recreated the project and attached it together with two logs from the P20Pro. One where there is no XR Provider set and one where ARCore is set.
I see that there is a camera error when transitioning from reconfiguring to capturing.
AnotherARProject.rar


Thank you for the help.

ARCoreXRProvider.txt NoXRPluginProvider.txt

Edited September 23, 2021 by RRob

[zzy]

Hi @RRob

Sorry for the late reply, I just got some time to test this.

Indeed the ARCore failed to start on my side. It seems that ARCore is taking more than 1 frame to start. Hand Tracking starts before ARCore and thus occupying the camera.

A quick way to fix this is that, you can disable GestureProvider script in your scene. You can add a script to wait until ARCore is started, and then enable GestureProvider script.

[kotauchisunsun]

Hi @zzy

I cannot use hand tracking and passthrough with AR Foundation, too. 

> Indeed the ARCore failed to start on my side. It seems that ARCore is taking more than 1 frame to start. Hand Tracking starts before ARCore and thus occupying the camera.
> quick way to fix this is that, you can disable GestureProvider script in your scene. You can add a script to wait until ARCore is started, and then enable GestureProvider script.

I disabled Gesture Provider script on ARFoundation.unty, and I added the ARSessionWatcher component to enable Gesture Provider script after initializing ARCore.
But, It's not work.

Unity 2020.3.10f1
AR Foundation 4.1.7
ARCore XR Plugin 4.1.7
Pixel 4
 

using System.Collections;

using System.Collections.Generic;

using UnityEngine;

using UnityEngine.XR.ARFoundation;

 

public class ARSessionWatcher : MonoBehaviour

{

    // Start is called before the first frame update

    public ARSessionState state;

    public int count = 0;

 

    void Start()

    {

        ARSession.stateChanged += (args) =>

        {

            state = args.state;

            Debug.Log("state: " + state.ToString());

            if (args.state == ARSessionState.SessionTracking)

            {

                Invoke("run", 5f);

            }

        };

    }

 

    void run()

    {

        Debug.Log("invoke " + count.ToString());

        count++;

        var component = GetComponent<ViveHandTracking.GestureProvider>();

        if (!component.enabled)

        {

            Debug.Log("enabled");

            component.enabled = true;

        }

    }

 

    // Update is called once per frame

    void Update()

    {

 

    }

}

 

arfoundation_with_vive_handtracking_20211030.txt that ARSessionWatcher is enabled.
arfoundation_without_arsessionwatcher_20211030.txt that ARSessionWatcher is disabled.

When ARSessionWatcher is enabled,  the application failed to create CameraStream (arfoundation_with_vive_handtracking_20211030.txt L202)

How to use hand tracking and pass through with AR Foundation?

arfoundation_without_arsessionwatcher_20211030.txt arfoundation_with_vive_handtracking_20211030.txt

[zzy]

Hi @kotauchisunsun

Thanks for providing the info. I have checked the arfoundation_with_vive_handtracking_20211030.txt. Here is related parts from the file:

10-30 16:42:53.092: I/Unity(6528): enabled
10-30 16:42:53.092: I/Unity(6528):  #0 0xbe6e5819 (libunity.so) ? 0x0
10-30 16:42:53.092: I/Unity(6528):  #1 0xbedaaa9f (libunity.so) ? 0x0
10-30 16:42:53.092: I/Unity(6528):  #2 0xbe8d55e5 (libunity.so) ? 0x0
10-30 16:42:53.092: I/Unity(6528):  #3 0xbe8d5551 (libunity.so) ? 0x0
10-30 16:42:53.092: I/Unity(6528):  #4 0xcd8c8da6 (Unknown) ? 0x0
10-30 16:42:53.134: I/Aristo(6528): Starting Vive Hand Tracking SDK 1.0.0 as 32-bit binary
10-30 16:42:53.148: I/Aristo(6528): Selected camera 0

I can see that after you enabled GestureProvider script, GestureProvider directly starts the detection with android camera. This means that is does not detect ARCore. Since camera is already used by ARCore, Vive Hand Tracking will fail to start.

However, there is something strange here. From GestureProvider.Start function (line 90-111), there should be logs that output engine selection process (no matter success or fail). But I cannot find any of these logs. So I think GestureProvider.Start may be already called before you enabled it from the script.

I would recommend you to check it in Unity Editor: 1) make sure GestureProvider is disabled in the scene 2) make sure GestureProvider script remains disabled after entered play mode in Unity Editor 3) you can manually enable GestureProvider from inspector and see if there are logs from GestureProvider.Start function in the console.

Another solution is that, you can completely remove GestureProvider in the scene and AddComponent in ARSessionWatcher.run function after ARCore is enabled.

[kotauchisunsun]

Hi @zzy

Thank you for reply. I tried it.

> Another solution is that, you can completely remove GestureProvider in the scene and AddComponent in ARSessionWatcher.run function after ARCore is enabled.

I delete GestureProvider Component in my scene. And, I modified the source code, ARSessionWatcher.cs. I made CheckGestureProvider function.
It checks to exist GestureProvider Component in Start(L14).  And, it checks to exist it when the script add the GestureProvider Component(L33) too. 

 But, Hand Tracking is not work.

I read the log, Track.txt. 
The log show GestureProvider Component is not exists on Start (L124).
And it show GestureProvider Component is not exists when the script add GestureProvider Component(L339) too.

> However, there is something strange here. From GestureProvider.Start function (line 90-111), there should be logs that output engine selection process (no matter success or fail). But I cannot find any of these logs. So I think GestureProvider.Start may be already called before you enabled it from the script.

In this log, I found "No suitable engine found" message. It's written by GestureProvider Component(L111). I configure the vive hand tracking settings below.
I think settings.Engines is not empty.

 

What should I try? 

ARSessionWatcher.cs Track.txt

[zzy]

Hi @kotauchisunsun

OK, from the log it seems the root cause is that vive hand tracking sdk doesn't detect arcore, thus it choose the normal android camera API to get camera stream. Since camera is already in use, the detection failed to start.

So the problem is inside code to detect arcore, which is defined in ViveHandTrackingEngine.cs (L195).

195> #if VIVEHANDTRACKING_UNITYXR_ARCORE && UNITY_ANDROID && !UNITY_EDITOR
196>  IEnumerator SetupARCore() {
197>    var settings = UnityEngine.XR.Management.XRGeneralSettings.Instance;
198>    if (settings == null) yield break;
199>    var manager = settings.Manager;
200>    if (manager == null || manager.activeLoader == null) yield break;
201>    var loader = manager.ActiveLoaderAs<UnityEngine.XR.ARCore.ARCoreLoader>();
202>    if (loader == null) yield break;

So we need to check 1) if this function is called 2) if this function returned early due to missing arcore loader.

So here is the check list:

1. The function is called at the beginning of StartDetection function (L100). Please check if VIVEHANDTRACKING_UNITYXR_ARCORE is defined. This is to make sure "com.unity.xr.arcore" package is installed and listed in package.json.
2. Before you add GestureProvider script in ARSessionWatcher.cs, you can check if settings, manager and loader are all not null. You can copy the code from L197-L202 and add logs to see if these variables are null. If any of these are null, you may try to wait for a few frames and retry.

Please only add Gestureprovider script after all the check passed to make sure hand tracking SDK knows arcore is running.

[hadysysdev]

Has anyone found a fix for this issue, currently facing the same issues