How to get output from the eye-tracker faster than Update in Unity

[imarin18]

Dear @jboss

What measurement parameter did you check for the interval? As posted in another thread, time_stamp parameter recorded by SRanipal SDK does not provide the value properly (HTC also recognises the issue). Otherwise, it would be better for you to try with another laptop that meets the requirements recommended by HTC (https://developer.vive.com/resources/vive-sense/hardware-guide/vive-pro-eye-specs-user-guide/).

Best regards,
imarin18

[jboss]

I found a good Black Friday deal, so I bought a new laptop. And now it looks like I get 8 ms intervals.

The values in the 100ns column increase by 40000 per row (which I think relates to 4 ms) and then I get the same SRanipal output, in two rows so that would be 8 ms.

I still will try to test this with Unity update rates higher than 120 fps, as I need that for my application. But I expect that the eyetracking will still run at 120Hz

[imarin18]

Dear @jboss

That's great to know you were able to confirm the sampling interval properly after changing the laptop. I suppose that the eye tracking will run at 120Hz, even if you change the FPS since the eye tracking should work independently from Unity fps configuration.

Best regards,
imarin18

 

[jboss]

Now I'm converting the saccade test project to my own project and I get new errors, so I hope someone can help me with those.

First I had to remove the static keyword from the eyecallback method, because I am logging the data and my CreateNewLogEntry method derives from a non-static base method. I can not judge the impact of that, but at least I got no compile errors.

Then I need to link the eyetracling log to my motion log where I log the ball and baseball bat. Since timestamp may have too much variation, I decided to use frame-number. But then I got the error that Time.frameCount can only be used in the main thread, so I cannot read that in the eyecallback thread. So I figured I should use eyedata.frame_sequence, but that has completely different values. Did a comparison with debug lines and found:
FixedUpdate in LoggerEyetracking at frame 31
EyeCallback called at eyedata.frame 276386, loggingActive=False

How can these vallues be so far apart

[jboss]

Well, screwed by technology again. Apparently I pressed some button to post this before it was fininshed and now I can not edit it anymore. So here's the full version.

Now I'm converting the saccade test project to my own project and I get new errors, so I hope someone can help me with those.

First I had to remove the static keyword from the eyecallback method, because I am logging the data and my CreateNewLogEntry method derives from a non-static base method. I can not judge the impact of that, but at least I got no compile errors.

Then I need to link the eyetracking log to my motion log where I log the ball and baseball bat, which is logged in Unity's fixedUpdate process, at 250 fps. Since timestamp may have too much variation, I decided to use frame-number. But then I got the error that Time.frameCount can only be used in the main thread, so I cannot read that in the eyecallback thread. So I figured I should use eyedata.frame_sequence, but that has completely different values. Did a comparison with debug lines and found:
      FixedUpdate in LoggerEyetracking at frame 31
      EyeCallback called at eyedata.frame 276386, loggingActive=False

How can these vallues be so far apart? And is there another way I can link the eyecallback thread to the frames in the main thread?

And every time there is an error the Unity editor freezes so I have to use Taskmanager to kill it and restart it.

But why does this have to be so complicated? This is costing me so much time that I can not spend on new content for my application.
Why can't it be connected to the FixedUpdate process in the main thread? Use 90fps and 90Hz as default (like it's now in VIU, link Physics Rate to Refresh Rate), and if I as developer change fixedDeltaTime to something between 90 and 120 the eyetracking frequency changes accordingly (or you can keep it at 90Hz, if that's too complicated). And when I set fixedDelateTime to anything higher than 120 the eyetracking frequency is 120Hz.

If it is running in the fixedUpdate proces, I can simply combine the motion log with the eyetracking data that is available at that moment, even if it may have been recorded 1 or 2 frames earlier.

[jboss]

I'm sorry that my last post was what rude. I just got taken by surprise that there was a time-limit on editing a post.

I'd really apreciate some help getting this working. Especially the freezing of the editor is killing me. After every test run I have to use task manager to end the editor and then start it again. That is not workable.

@imarin18
I checked what happens in your application at the end becase that application ends normally and copied that to a Coroutine in my application (see code below) and then use Update to call this Coroutine when the X key is pressed. I can see in the log that the Coroutine is executed, but still the editor freezes.

    IEnumerator EndEyeTracking()
    {
        // Stuff from saccade test project that is done at the end of the test
        //debugWindow.text = "Test finished. Waiting " + Endbuffer.ToString() + " seconds";

        yield return new WaitForSeconds(Endbuffer);

        SaccadeEndTime = DateTime.Now.Ticks;

        // JB changed because original code causes build errors
        if (Application.isPlaying)
        {
            Application.Quit();
        }
        Debug.Log("Eyetracking finished");
    }

I know that in the editor Application.IsPlaying is false, so the application does not quit by itself. Instead I get the debug line below it.

In the other forum thread that mentions the freezing of the editor there are several options and I get confused which is which. One uses new Thread() and then use OnApplicationQuit and OnDisable() to abort the eyetracking, but in the saccade test this is not used.

https://forum.vive.com/topic/5897-getting-verbosedata-at-the-fastest-rate-possible/page/2/

So I hope someone can help me with this. I attached the editor.log file, maybe that helps.

 

Editor_Freeze_OnePitch.log

[imarin18]

Dear @jboss

The reason why you face the frozen Unity is that the eye tracker is still measureing the eye movements, while the process in void Update () is completed. As long as you continue to record the data of eye movements, Unity playing will not end. The GitHub page says as below.

"Please be noted that you will see Unity seem to be frozen if you click the stop button on Unity for a specific case. Unity seems to be frozen in this situation because the sampling of eye tracking is still working on the background. As explained in the following algorithm flow chart, the sampling of eye tracking works independently from Unity. The playing of Unity is stopped automatically when all the saccade tasks are completed. We set the maximum sampling number of eye tracking to 120 Hz * 1800 seconds in line# 175: private const int maxframe_count = 120 * 1800; of C# script: Saccade_measure_rev1.cs. Therefore, if you click the stop button on Unity within maxframe_count time range, Unity seems frozen (actually not because the system still waits for the sampling of eye tracking to finish). On the other hand, if you click the stop button beyond maxframe_count time range, the playing of Unity stops immediately. The maximum number of maxframe_count can be adjusted accordingly." 

If you adjust the value of private const int maxframe_count on the script according to your preference, I suppose you can avoid the frozen Unity.

Best regards,
imarin18 

[jboss]

Hi @imarin18

I realise I forgot to mention something. I made a small adaptation to the script. In the while loops in the Coroutine Sequence I added a check for a boolean called taskInterrupted. This boolean is set when I press the application menu button on the Vive controller. When that boolean is true this breaks the while loop showing the targets. And then the last part of the Sequence Coroutine is executed and there saccadeEndTime is set. And that actually also breaks the while loop in the EyeCallback method, I see now. Even though cnt_callback < maxframe_count.

        // ======================================================================
        //  1. Perform saccadic eye movement assessment: 60 pro-saccade trials.
        // ======================================================================
        Debug.Log("1st pro-saccade test has started.");
        debugWindow.text="1st pro-saccade test has started."; // JB last step for testing, no changes in the rest of the procedure
        for (int i = 0; i < pro_n; i++)
        {
            yield return StartCoroutine(TargetAppear(prosac_time_1[i], prosac_direct_1[i], i));
            if (taskInterrupted)
            {
                break;
            }
        }

        /* Skip rest of the test
        Debug.Log("Take 1 minute break. Press space key to start 4 practice trials of anti-saccade task.");

So now that I write this answer I realise that the problem must be in how I stop the eyetracking in my application. And I understand better how your application works as well. 🙂

So thanks for your reaction, 

[imarin18]

Dear @jboss

Thank you very much for your response.

I think your understanding is correct. We need to remember that Unity playing does not always work along with eye tracking. In your script posted previously, you wrote the code below. This stops only the playing of Unity, not eye tracking. Thus, if you want to stop the eye tracking in parallel, one of the workarounds would be to use UnixTime (i.e. SaccadeEndTime) as a trigger to stop the eye tracking, as designed in the GitHub page: https://github.com/MotorControlLearning/SaccadeVR-mobile.

 if (Application.isPlaying)
        {
            Application.Quit();
        }

Best regards,

imarin18

[maple]

On 10/9/2019 at 7:56 PM, Daniel_Y said:

There is callback mode supported in 120 FPS now after check "Enable Eye Data Callback" as the figure shown below; And, then register a callback function by SRanipal_Eye_v2.WrapperRegisterEyeDataCallback(). Could refer to its usage in sample included in SDK.

 

 

I chosed according to this, but my version is SRanipal_GazeRaySample.cs.Using this i could get data in 90HZ,if i use SRanipal_GazeRaySample_v2.cs,i just got data in 50HZ.Why does it lead to such a structure?

Thanks in advance!