Tobii 4C – Electron integration with core SDK and Tobii Interaction 0.7.3

[Rob]

I found this useful github repo that enabled me to stream Tobii 4c gaze data to an nodejs electron app. I have managed to set this up and get it functional. The electron application opens in full screen and I created a full screen canvas and have been printing out the gaze points on the canvas. The coordinates are way off though. I know that Tobii provides the gaze data in mm not px, which makes this more complicated. So I tried to convert the mm to px with special libraries and custom written code, however it did not work at all. The libraries pushed the eye tracking data way off the screen. When I divided the gaze data by 1.25, that was sent from Tobii C# server, it was the closest thing to accurate positioning. Problem is that I calculated how many pixels there are per mm and turns out to be 4 on my current screen resolution. So I would have assumed that I would have to multiply the mm that come back from the server times 4. That would send the result way off the screen.

I have been reading through all the SDK’s and struggling to figure out how I can pass on the screen dimensions onto my electron app, to ensure that the display size of Tobii matches the application size. Any suggestions?

https://github.com/frocker/tobii-electron-streaming

[Grant [Tobii]]

Hi @rockyhub, thanks for your query. The coordinates of the Gaze Point data stream and the Fixation data stream are provided in (x,y) points in pixels on the Virtual Screen. This is how the Virtual Screen is described on MSDN: https://msdn.microsoft.com/en-us/library/dd145136(v=vs.85).aspx

Are you using some other stream modifier to receive the values in mm?

In any case, since (0,0) is the upper left corner of the primary screen, whenever you look to the left or above the primary screen’s borders you will get negative values for the x and/or y coordinate in these EyeX data streams.

Accordingly, to convert the gaze values to pixels it should be sufficient to multiply the normalised (0,1) gaze co-ordinates by the pixel length of the screen which you can access via a regular .NET library.

For example,

Display display = getWindowManager().getDefaultDisplay();
Point size = new Point();
display.getSize(size);
int width = size.x;
int height = size.y;

Hopefully, this answers your query, but please let us know if we can be of any further assistance.

[Rob]

Hi Grant,

Your suggestions make sense assuming I have access to the normalised data in the Core SDK.

From what I read up online, the Tobii Core SDK doesn’t provide normalized data for the Gaze Point data stream. The EyeX used to. Am I incorrect in that assumption? If I download and install https://www.nuget.org/packages/Tobii.EyeX.Client/, will I be able to access the normalised coordinates?

using System;
using System.Net.Sockets;
using System.Text;
using Tobii.Interaction;
using Tobii.Interaction.Framework;
using Tobii.Interaction.Model;

/*Integrate this into a C# project that is set up to use the Tobii SDK.
  It will send the Tobii input to your electron App over UDP.
*/
namespace TobiiSDKServer
{
    class TobiiServer
    {
        static void Main(string[] args)
        {

            // Initialise Host to Tobii Connection
            var host = new Host();

            //Uncomment this section to Launch Calibration when the project opens
            /*
             System.Threading.Thread.Sleep(1000);
             host.Context.LaunchConfigurationTool(ConfigurationTool.RetailCalibration, (data) => { });
             System.Threading.Thread.Sleep(10000);
            */

            //Setup Server
            UdpClient udpClient = new UdpClient();
            udpClient.Connect("127.0.0.1", 33333);

            //Create stream. 
            var gazePointDataStream = host.Streams.CreateGazePointDataStream();

            // Get the gaze data
            gazePointDataStream.GazePoint((x, y, ts) => SendInput(udpClient, x, y, ts));

            // Read
            Console.ReadKey();

            // we will close the coonection to the Tobii Engine before exit.
            host.DisableConnection();

            //ToDo: Add code to boot your Electron App here

        }

        static void SendInput(UdpClient client, double x, double y, double ts)
        {
            String sendString = @"{""id"":""gaze_data"", ""x"":" + x + @", ""y"": " + y + @", ""timestamp"":" + ts + @"}";
            Console.WriteLine(sendString);
            Byte[] senddata = Encoding.ASCII.GetBytes(sendString);
            client.Send(senddata, senddata.Length);
        }
    }
}

I have done some modifications to this code on my other computer, though from what I can tell the results coming back from this are in mm and not pixels and I am struggling to get the normalised gazed coordinates.

[Rob]

To answer your question, I do not believe I am using any other stream modifier.

[Grant [Tobii]]

Hi @rockyhub,

From what I read up online, the Tobii Core SDK doesn’t provide normalized data for the Gaze Point data stream. The EyeX used to. Am I incorrect in that assumption?

Yes, indeed the gaze data as provided by the Core SDK is normalised to screen co-ordinates by default.

For simplicity, if you check out the Core SDK “Streams” samples @ https://github.com/Tobii/CoreSDK/tree/master/samples/Streams

Therein you will see how the various values such as gaze data, head tracking etc are reported by the SDK.

I am not sure how the latest version of the Core SDK will operate with your electron project, but first try out the streams sample and thereafter introduce your electron code which hopefully should still work ok. Let us know how you get on. Thanks.

[Rob]

Hi Grant,

Thank you! Got it working well. Got it working to draw on the canvas in the right places.

Now I have another very random issue. It was working fine and then all of a sudden the tracker stopped working. It enables me to calibrate the device and activate eye tracking for that purpose and then once calibration has finished it says “eye tracking not available” and in the Components area it says that the Tobii Controller Core is not available. Any suggestions as to why this happened and so randomly?

[Grant [Tobii]]

Hi @rockyhub, glad to hear you got it working.

With regard to connectivity issues, please try going through the steps outlines below which usually resolves most problems of this nature

https://help.tobii.com/hc/en-us/articles/115000436349-Is-your-Eye-Tracker-not-connecting-

I would recommend going for a fresh uninstall and reinstall of latest Core Sofware first.

Thereafter, please let us know how you get on. Thanks.

[Author]

Posts