Android hand tracking - starting tips ?

[VirtualPierogi]

Thats probably not the case as I have both the controller and daydream view on my desk, but I've noticed the following problem in adb log :

05-29 12:03:26.071   948  1005 I Unity   : SystemInfo CPU = ARMv7 VFPv3 NEON, Cores = 8, Memory = 3679mb05-29 12:03:26.071   948  1005 I Unity   : SystemInfo ARM big.LITTLE configuration: 4 big (mask: 0xf0), 4 little (mask: 0xf)05-29 12:03:26.073   948  1005 I Unity   : ApplicationInfo com.XRInstitute.HandTracking version 0.1 build 5b1ec8d0-32fe-4103-88e1-ac25ff79496405-29 12:03:26.073   948  1005 I Unity   : Built from '2019.1/staging' branch, Version '2019.1.4f1 (ffa3a7a2dd7d)', Build type 'Release', Scripting Backend 'mono', CPU 'armeabi-v7a', Stripping 'Disabled'05-29 12:03:26.456   948  1005 E Unity   : Unable to find libaudiopluginresonanceaudio--------- beginning of system05-29 12:03:26.825   948  1005 D Unity   :  GL_EXT_debug_marker GL_ARM_rgba8 GL_ARM_mali_shader_binary GL_OES_depth24 GL_OES_depth_texture GL_OES_depth_texture_cube_map GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_EXT_read_format_bgra GL_OES_compressed_paletted_texture GL_OES_compressed_ETC1_RGB8_texture GL_OES_standard_derivatives GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_EGL_sync GL_OES_texture_npot GL_OES_vertex_half_float GL_OES_required_internalformat GL_OES_vertex_array_object GL_OES_mapbuffer GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_type_2_10_10_10_REV GL_OES_fbo_render_mipmap GL_OES_element_index_uint GL_EXT_shadow_samplers GL_OES_texture_compression_astc GL_KHR_texture_compression_astc_ldr GL_KHR_texture_compression_astc_hdr GL_KHR_texture_compression_astc_sliced_3d GL_KHR_debug GL_EXT_occlusion_query_boolean GL_EXT_disjoint_timer_query GL_EXT_blend_minmax GL_EXT_discard_framebuffer GL_OES_get_program_binary GL_OES_texture_3D GL_EXT_texture_storage GL_EXT_multisampled_render_05-29 12:03:26.825   948  1005 D Unity   : to_texture GL_EXT_multisampled_render_to_texture2 GL_OES_surfaceless_context GL_OES_texture_stencil8 GL_EXT_shader_pixel_local_storage GL_ARM_shader_framebuffer_fetch GL_ARM_shader_framebuffer_fetch_depth_stencil GL_ARM_mali_program_binary GL_EXT_sRGB GL_EXT_sRGB_write_control GL_EXT_texture_sRGB_decode GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_RG8 GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent GL_OES_texture_storage_multisample_2d_array GL_OES_shader_image_atomic GL_EXT_robustness GL_EXT_draw_buffers_indexed GL_OES_draw_buffers_indexed GL_EXT_texture_border_clamp GL_OES_texture_border_clamp GL_EXT_texture_cube_map_array GL_OES_texture_cube_map_array GL_OES_sample_variables GL_OES_sample_shading GL_OES_shader_multisample_interpolation GL_EXT_shader_io_blocks GL_OES_shader_io_blocks GL_EXT_tessellation_shader GL_OES_tessellation_shader GL_EXT_primitive_bounding_box GL_OES_primitive_bounding_box GL_EXT_geometry_shader GL_OES_geometry_shader GL_ANDROID_extension_pack_es31a GL_EXT_gpu_sh05-29 12:03:26.825   948  1005 D Unity   : ader5 GL_OES_gpu_shader5 GL_EXT_texture_buffer GL_OES_texture_buffer GL_EXT_copy_image GL_OES_copy_image GL_EXT_shader_non_constant_global_initializers GL_EXT_color_buffer_half_float GL_EXT_color_buffer_float GL_EXT_YUV_target GL_OVR_multiview GL_OVR_multiview2 GL_OVR_multiview_multisampled_render_to_texture GL_KHR_robustness GL_KHR_robust_buffer_access_behavior GL_EXT_draw_elements_base_vertex GL_OES_draw_elements_base_vertex GL_EXT_protected_textures GL_EXT_buffer_storage GL_EXT_external_buffer GL_EXT_EGL_image_array05-29 12:03:29.324   948  1005 I Unity   : Created eye textures with a "separate" layout.  The "multi-pass" stereo mode will be used.05-29 12:03:29.324   948  1005 I Unity   :05-29 12:04:00.229   948  1005 I Unity   : GVR SDK for Unity version: 1.200.005-29 12:04:00.229   948  1005 I Unity   :05-29 12:04:00.229   948  1005 I Unity   : (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 48)05-29 12:04:00.229   948  1005 I Unity   :05-29 12:04:00.557   948  1005 I Aristo  : Running as 32-bit binary05-29 12:04:00.663   948  1005 I Aristo  : Selected camera 005-29 12:04:02.783   948  2109 E Aristo  : Android NDK camera update frame failed!05-29 12:04:02.783   948  2109 E Aristo  : Camera update failed, stop detection

It seems that the android ndk is failing to update camera, I dont have NDK installed but will try with it next, do I have to change from mono to il2cpp ?

[zzy]

Hi  

 

You are right, the problem is in Hand Tracking SDK now, since we cannot get camera frames from your rear camera. From the log, we can find a rear camera, but cannot get frames from it.

 

If you are building 32bit binary, using mono is definitely fine.

 

Please make sure:

1. Your app have camera permission (I recommend you grant permission manually in Settings)

2. Some phones may not support NDK Camera2 API, which is what we use to get frames. A simple check is to try this example and see if it works.

[VirtualPierogi]

Allright, the example Native Android Camera2 shows black, is there any option to enable it or change the api from camera2 to other camera provider?
edit:

After a while it seems that native android camera actually does work, so that makes me wonder what doesnt work on the aristo side.

[zzy]

OK, it seems that on your phone, it takes a longer time before the camera streaming starts to output frames. We internally have a 1.5 second timeout for waiting one frame. This seems to be the reason. I'll try to increase the timeout for the first frame in next version (expected to release in a few days).

[VirtualPierogi]

Thanks for Your help, ill keep in touch for the nex version, 

[MDV Games]

 

In this release will available Unreal Plugin for Vive Focus?

[zzy]

Hi , yes the next version includes Android phone and Focus support in Unreal plugin

[MDV Games]

 

Great! Thanks!

[zzy]

Hi & , please check SDK new release v0.8.1.

[VirtualPierogi]

Hey,
I've tried the update and there is a progress - it actually started!
I can see the green and blue balls flying where my hand should be, thats unbeliveable.

And it recognizes gestures too! for now only pointing fist and all five works, "ok" gesture didnt trigger nothing yet.

Phone gets hot pretty quick.

And it seem to lose track of hand centre and jump to fingers but thats ok.

Sometimes hand tracking stops and doesnt want to start but after app forced close it worked again, 

Implying that Aristo service runs after app was closed or stays in memory.

Anyway im pretty excited for this technology to mature and ways I could use it!

 

Dropping logcat for good measure:

PS C:\Users\USER> adb logcat -s Unity Aristo--------- beginning of crash--------- beginning of main06-01 17:30:43.501 26103 26125 I Unity   : SystemInfo CPU = ARMv7 VFPv3 NEON, Cores = 8, Memory = 3679mb06-01 17:30:43.501 26103 26125 I Unity   : SystemInfo ARM big.LITTLE configuration: 4 big (mask: 0xf0), 4 little (mask: 0xf)06-01 17:30:43.502 26103 26125 I Unity   : ApplicationInfo com.XRInstitute.HandTracking version 0.1 build db70123d-ea30-48e1-90d4-ca22b1ff63dc06-01 17:30:43.502 26103 26125 I Unity   : Built from '2019.1/staging' branch, Version '2019.1.4f1 (ffa3a7a2dd7d)', Build type 'Release', Scripting Backend 'mono', CPU 'armeabi-v7a', Stripping 'Disabled'06-01 17:30:43.843 26103 26125 E Unity   : Unable to find libaudiopluginresonanceaudio06-01 17:30:43.908 26103 26125 D Unity   :  GL_EXT_debug_marker GL_ARM_rgba8 GL_ARM_mali_shader_binary GL_OES_depth24 GL_OES_depth_texture GL_OES_depth_texture_cube_map GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_EXT_read_format_bgra GL_OES_compressed_paletted_texture GL_OES_compressed_ETC1_RGB8_texture GL_OES_standard_derivatives GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_EGL_sync GL_OES_texture_npot GL_OES_vertex_half_float GL_OES_required_internalformat GL_OES_vertex_array_object GL_OES_mapbuffer GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_type_2_10_10_10_REV GL_OES_fbo_render_mipmap GL_OES_element_index_uint GL_EXT_shadow_samplers GL_OES_texture_compression_astc GL_KHR_texture_compression_astc_ldr GL_KHR_texture_compression_astc_hdr GL_KHR_texture_compression_astc_sliced_3d GL_KHR_debug GL_EXT_occlusion_query_boolean GL_EXT_disjoint_timer_query GL_EXT_blend_minmax GL_EXT_discard_framebuffer GL_OES_get_program_binary GL_OES_texture_3D GL_EXT_texture_storage GL_EXT_multisampled_render_06-01 17:30:43.908 26103 26125 D Unity   : to_texture GL_EXT_multisampled_render_to_texture2 GL_OES_surfaceless_context GL_OES_texture_stencil8 GL_EXT_shader_pixel_local_storage GL_ARM_shader_framebuffer_fetch GL_ARM_shader_framebuffer_fetch_depth_stencil GL_ARM_mali_program_binary GL_EXT_sRGB GL_EXT_sRGB_write_control GL_EXT_texture_sRGB_decode GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_RG8 GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent GL_OES_texture_storage_multisample_2d_array GL_OES_shader_image_atomic GL_EXT_robustness GL_EXT_draw_buffers_indexed GL_OES_draw_buffers_indexed GL_EXT_texture_border_clamp GL_OES_texture_border_clamp GL_EXT_texture_cube_map_array GL_OES_texture_cube_map_array GL_OES_sample_variables GL_OES_sample_shading GL_OES_shader_multisample_interpolation GL_EXT_shader_io_blocks GL_OES_shader_io_blocks GL_EXT_tessellation_shader GL_OES_tessellation_shader GL_EXT_primitive_bounding_box GL_OES_primitive_bounding_box GL_EXT_geometry_shader GL_OES_geometry_shader GL_ANDROID_extension_pack_es31a GL_EXT_gpu_sh06-01 17:30:43.908 26103 26125 D Unity   : ader5 GL_OES_gpu_shader5 GL_EXT_texture_buffer GL_OES_texture_buffer GL_EXT_copy_image GL_OES_copy_image GL_EXT_shader_non_constant_global_initializers GL_EXT_color_buffer_half_float GL_EXT_color_buffer_float GL_EXT_YUV_target GL_OVR_multiview GL_OVR_multiview2 GL_OVR_multiview_multisampled_render_to_texture GL_KHR_robustness GL_KHR_robust_buffer_access_behavior GL_EXT_draw_elements_base_vertex GL_OES_draw_elements_base_vertex GL_EXT_protected_textures GL_EXT_buffer_storage GL_EXT_external_buffer GL_EXT_EGL_image_array06-01 17:30:46.089 26103 26125 I Unity   : Created eye textures with a "separate" layout.  The "multi-pass" stereo mode will be used.06-01 17:30:46.089 26103 26125 I Unity   :06-01 17:31:01.528 26103 26125 I Unity   : GVR SDK for Unity version: 1.200.006-01 17:31:01.528 26103 26125 I Unity   :06-01 17:31:01.528 26103 26125 I Unity   : (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 48)06-01 17:31:01.528 26103 26125 I Unity   :06-01 17:31:02.225 26103 26125 E Unity   : AndroidJavaException: java.lang.NullPointerException: Attempt to invoke virtual method 'int android.app.Activity.checkSelfPermission(java.lang.String)' on a null object reference06-01 17:31:02.225 26103 26125 E Unity   : java.lang.NullPointerException: Attempt to invoke virtual method 'int android.app.Activity.checkSelfPermission(java.lang.String)' on a null object reference06-01 17:31:02.225 26103 26125 E Unity   :      at com.google.gvr.permissionsupport.PermissionsFragment.hasPermission(PermissionsFragment.java:88)06-01 17:31:02.225 26103 26125 E Unity   :      at com.unity3d.player.UnityPlayer.nativeRender(Native Method)06-01 17:31:02.225 26103 26125 E Unity   :      at com.unity3d.player.UnityPlayer.c(Unknown Source:0)06-01 17:31:02.225 26103 26125 E Unity   :      at com.unity3d.player.UnityPlayer$e$1.handleMessage(Unknown Source:88)06-01 17:31:02.225 26103 26125 E Unity   :      at android.os.Handler.dispatchMessage(Handler.java:102)06-01 17:31:02.225 26103 26125 E Unity   :      at android.os.Looper.loop(Looper.java:214)06-01 17:31:02.225 26103 26125 E Unity   :      at com.unity3d.player.UnityPlayer$e.run(Unknown Source:20)06-01 17:31:02.225 26103 26125 E Unity   :   at UnityEngine.AndroidJNISafe.CheckException () [0x00091] in <549ed8055ed14737ac3b841d26182fec>:006-01 17:31:02.225 26103 26125 E Unity   :   at UnityEngine.AndroidJNISafe.CallBooleanMethod (System.IntPtr obj, System.IntPtr methodID, UnityEngine.jvalue[] args) [0x00011] in <06-01 17:32:29.053 27650 27670 I Unity   : SystemInfo CPU = ARMv7 VFPv3 NEON, Cores = 8, Memory = 3679mb06-01 17:32:29.053 27650 27670 I Unity   : SystemInfo ARM big.LITTLE configuration: 4 big (mask: 0xf0), 4 little (mask: 0xf)06-01 17:32:29.054 27650 27670 I Unity   : ApplicationInfo com.XRInstitute.HandTracking version 0.1 build db70123d-ea30-48e1-90d4-ca22b1ff63dc06-01 17:32:29.054 27650 27670 I Unity   : Built from '2019.1/staging' branch, Version '2019.1.4f1 (ffa3a7a2dd7d)', Build type 'Release', Scripting Backend 'mono', CPU 'armeabi-v7a', Stripping 'Disabled'06-01 17:32:29.308 27650 27670 E Unity   : Unable to find libaudiopluginresonanceaudio06-01 17:32:29.398 27650 27670 D Unity   :  GL_EXT_debug_marker GL_ARM_rgba8 GL_ARM_mali_shader_binary GL_OES_depth24 GL_OES_depth_texture GL_OES_depth_texture_cube_map GL_OES_packed_depth_stencil GL_OES_rgb8_rgba8 GL_EXT_read_format_bgra GL_OES_compressed_paletted_texture GL_OES_compressed_ETC1_RGB8_texture GL_OES_standard_derivatives GL_OES_EGL_image GL_OES_EGL_image_external GL_OES_EGL_image_external_essl3 GL_OES_EGL_sync GL_OES_texture_npot GL_OES_vertex_half_float GL_OES_required_internalformat GL_OES_vertex_array_object GL_OES_mapbuffer GL_EXT_texture_format_BGRA8888 GL_EXT_texture_rg GL_EXT_texture_type_2_10_10_10_REV GL_OES_fbo_render_mipmap GL_OES_element_index_uint GL_EXT_shadow_samplers GL_OES_texture_compression_astc GL_KHR_texture_compression_astc_ldr GL_KHR_texture_compression_astc_hdr GL_KHR_texture_compression_astc_sliced_3d GL_KHR_debug GL_EXT_occlusion_query_boolean GL_EXT_disjoint_timer_query GL_EXT_blend_minmax GL_EXT_discard_framebuffer GL_OES_get_program_binary GL_OES_texture_3D GL_EXT_texture_storage GL_EXT_multisampled_render_06-01 17:32:29.398 27650 27670 D Unity   : to_texture GL_EXT_multisampled_render_to_texture2 GL_OES_surfaceless_context GL_OES_texture_stencil8 GL_EXT_shader_pixel_local_storage GL_ARM_shader_framebuffer_fetch GL_ARM_shader_framebuffer_fetch_depth_stencil GL_ARM_mali_program_binary GL_EXT_sRGB GL_EXT_sRGB_write_control GL_EXT_texture_sRGB_decode GL_EXT_texture_sRGB_R8 GL_EXT_texture_sRGB_RG8 GL_KHR_blend_equation_advanced GL_KHR_blend_equation_advanced_coherent GL_OES_texture_storage_multisample_2d_array GL_OES_shader_image_atomic GL_EXT_robustness GL_EXT_draw_buffers_indexed GL_OES_draw_buffers_indexed GL_EXT_texture_border_clamp GL_OES_texture_border_clamp GL_EXT_texture_cube_map_array GL_OES_texture_cube_map_array GL_OES_sample_variables GL_OES_sample_shading GL_OES_shader_multisample_interpolation GL_EXT_shader_io_blocks GL_OES_shader_io_blocks GL_EXT_tessellation_shader GL_OES_tessellation_shader GL_EXT_primitive_bounding_box GL_OES_primitive_bounding_box GL_EXT_geometry_shader GL_OES_geometry_shader GL_ANDROID_extension_pack_es31a GL_EXT_gpu_sh06-01 17:32:29.398 27650 27670 D Unity   : ader5 GL_OES_gpu_shader5 GL_EXT_texture_buffer GL_OES_texture_buffer GL_EXT_copy_image GL_OES_copy_image GL_EXT_shader_non_constant_global_initializers GL_EXT_color_buffer_half_float GL_EXT_color_buffer_float GL_EXT_YUV_target GL_OVR_multiview GL_OVR_multiview2 GL_OVR_multiview_multisampled_render_to_texture GL_KHR_robustness GL_KHR_robust_buffer_access_behavior GL_EXT_draw_elements_base_vertex GL_OES_draw_elements_base_vertex GL_EXT_protected_textures GL_EXT_buffer_storage GL_EXT_external_buffer GL_EXT_EGL_image_array06-01 17:32:31.092 27650 27670 I Unity   : Created eye textures with a "separate" layout.  The "multi-pass" stereo mode will be used.06-01 17:32:31.092 27650 27670 I Unity   :06-01 17:32:49.240 27650 27670 I Unity   : GVR SDK for Unity version: 1.200.006-01 17:32:49.240 27650 27670 I Unity   :06-01 17:32:49.240 27650 27670 I Unity   : (Filename: ./Runtime/Export/Debug/Debug.bindings.h Line: 48)06-01 17:32:49.240 27650 27670 I Unity   :06-01 17:32:49.896 27650 27670 E Unity   : AndroidJavaException: java.lang.NullPointerException: Attempt to invoke virtual method 'int android.app.Activity.checkSelfPermission(java.lang.String)' on a null object reference06-01 17:32:49.896 27650 27670 E Unity   : java.lang.NullPointerException: Attempt to invoke virtual method 'int android.app.Activity.checkSelfPermission(java.lang.String)' on a null object reference06-01 17:32:49.896 27650 27670 E Unity   :      at com.google.gvr.permissionsupport.PermissionsFragment.hasPermission(PermissionsFragment.java:88)06-01 17:32:49.896 27650 27670 E Unity   :      at com.unity3d.player.UnityPlayer.nativeRender(Native Method)06-01 17:32:49.896 27650 27670 E Unity   :      at com.unity3d.player.UnityPlayer.c(Unknown Source:0)06-01 17:32:49.896 27650 27670 E Unity   :      at com.unity3d.player.UnityPlayer$e$1.handleMessage(Unknown Source:88)06-01 17:32:49.896 27650 27670 E Unity   :      at android.os.Handler.dispatchMessage(Handler.java:102)06-01 17:32:49.896 27650 27670 E Unity   :      at android.os.Looper.loop(Looper.java:214)06-01 17:32:49.896 27650 27670 E Unity   :      at com.unity3d.player.UnityPlayer$e.run(Unknown Source:20)06-01 17:32:49.896 27650 27670 E Unity   :   at UnityEngine.AndroidJNISafe.CheckException () [0x00091] in <549ed8055ed14737ac3b841d26182fec>:006-01 17:32:49.896 27650 27670 E Unity   :   at UnityEngine.AndroidJNISafe.CallBooleanMethod (System.IntPtr obj, System.IntPtr methodID, UnityEngine.jvalue[] args) [0x00011] in <--------- beginning of system