Tobii EyeX is a complete eye tracking solution that includes both hardware and software to create the Tobii Eye Experience: natural human-computer interactions using a combination of eye-gaze and other user input.

Tobii_Eye_Experience_EyeXBox_297x185

The Tobii EyeX Controller is an eye tracking device that uses near-infrared light to track the eye movements and gaze point of a user.

The Tobii EyeX Engine is a piece of software that works like an OS extension for eye tracking. It knows how to configure and talk to the eye tracker, react to system changes, combine the user’s eye-gaze and other input and interpret them as user interactions, and it mediates between multiple applications that are using eye tracking simultaneously. It also contains a whole lot of eye tracking smartness harnessed over the 10+ years of eye tracking experience at Tobii. This is offered as filtered data streams and built-in heuristics that are specialized in figuring out what the user is actually looking at.

Tobii EyeX Interaction is a piece of software built on top of the EyeX Engine, and offers a set of basic eye-gaze interactions available out-of-the-box in the Windows environment.

The EyeX SDK provides you with access to the EyeX Engine API. It includes code samples, demo scenes, dlls, documentation and code for integration into a selected set of game engines and GUI frameworks.

Streams, states and behaviors

The EyeX Engine API can basically be divided into three categories: Streams, States and Behaviors.

The Streams provide nicely filtered eye-gaze data from the eye tracker transformed to a convenient coordinate system. The point on the screen where your eyes are looking (gaze point), and the points on the screen where your eyes linger to focus on something (fixations) are given as pixel coordinates on the screen. The positions of your eyeballs (eye positions) are given in space coordinates in millimeters relative to the center of the screen.

The States provide information about the current state of the EyeX system. There are a couple of more dynamic states related to the user: the user presence state (if there is a user in front of the screen or not) and the gaze tracking state (if the eye tracker is currently able to track the user’s eye-gaze or not). There are also states that are related to the system setup, such as the size in millimeters of the display the eye tracker is currently setup for, the currently selected user calibration profile, or if the eye tracker is currently available.

The Behaviors are higher level interaction concepts. You can compare them with familiar mouse interactions like click and scroll. The interaction behaviors are related to regions on the screen rather than single points. The regions typically correspond to GUI components like clickable buttons or scrollable panels. The concept of a gaze-aware region is a region that knows if a user is looking at it or not. The EyeX Engine uses advanced heuristics and filtering to decide which object a user is trying to interact with and how. This gives a much more stable and precise interaction than one that simply uses the coordinate of the latest gaze point.

How the EyeX Engine knows what the user is looking at

  1. The user looks at the screen. The EyeX Controller calculates the coordinates of the user’s gaze point. The EyeX Engine receives the normalized screen coordinates and transforms them to pixel coordinates on the screen.

    EyeXControllerGazePointCoordinates

  2. The EyeX Engine uses information from the client application to understand what kind of visual object the user is looking at and might want to interact with. Let’s say the rectangles on the screen are buttons with the ‘Activatable’ behavior. That means that they can be clicked using eye gaze. The EyeX Engine now decides which button the user is looking at and then expects the user to indicate that he or she wants to click the button.
  3. The client application might offer the user some different ways of actually clicking the button using eye-gaze, but it will always be some combination of looking at the thing you want to click, and then giving some secondary input to trigger the click. Let’s say that in this client application the user can trigger a click on the object they are looking at by pressing the space bar on the keyboard.

    Combine eye and key press to interact at gaze point

  4. The user presses the space bar while looking at the button he or she wants to click. The client application informs the EyeX Engine that the user has requested to ‘Activate’ the activatable button currently looked at.
  5. The EyeX Engine generates an event that informs the client application which button the user wants to activate.

    ActivationEvent

  6. The client application responds to the event and for example gives the user a visual feedback that the button on the screen is clicked, and then performs the action associated with clicking that button.

    ApplicationHandlesEvent

The Windows Presentation Foundation (WPF) and WinForms integrations included in the EyeX SDK for .Net makes a lot of the steps above seamless. The app developer specifies the desired EyeX behavior for a certain GUI element, and write the click handling code. The rest is taken care of by client side code included in the SDK.