Gaze To Object Mapping - Tobii Developer Zone

Gaze To Object Mapping (GTOM) is a process that takes gaze data as input and outputs the object, or objects, that the process deems the user is looking at. This is vital for virtually all real-life applications of eye tracking. There are of course some exceptions, for example some implementations of foveated rendering (Improving rendering performance or quality using gaze data) and Infinite Screen. It is worth noting even Infinite Screen/Extended View does not use the raw gaze data however, but rather uses a heavily processed data stream.

Why you should use GTOM

While it is tempting to use the raw gaze data, we strongly recommend you resist that urge. We have spent many years working on, and perfecting GTOM at this point, so please leverage that body of knowledge.

How to use GTOM

Using GTOM in UE4 is quite simple. You just need to do two things to get started. First you need a “Gaze Focus Manager”. We recommend adding one of these components to your player controller, but you are free to put it whereever you want. This will be your access point for getting focus data. After this is set up, you just need to mark the objects in the world you want to be gaze focusable. There are two ways you can do this in UE4.

GTOM Component tags

The first one is using component tags. Just adding a tag called “GazeFocusable” to any primitive component will make that component participate in GTOM. You can also customize some aspects of GTOM using other tags. All of them are listed in TobiiGTOMTypes.h, or you can see them here:

class TOBIIGTOM_API FTobiiGazeFocusTags
{
public:
    static FName HasGazeFocusTag;                       //A primitive component with this tag currently has gaze focus.
    static FName GazeFocusableTag;                      //A primitive component with this tag will participate in GTOM even if the actor owning the primitive doesn't have a GazeFocusableComponent, or if the default for the gaze focusable is off.
    static FName NotGazeFocusableTag;                   //A primitive component with this tag will not participate in GTOM even if the actor owning the primitive component has a GazeFocusableComponent.
    static FString GazeFocusablePriorityTag;            //A primitive component with this tag will modify it's priority. Priority is used when only a certain number of focusables can participate in an operation. An example of this is ID buffer based GTOM. This is an argument tag, this means you must provide the value after the tag. Usage example: GazeFocusablePriorityTag 100
    static FString GazeFocusableMaximumDistanceTag;     //A primitive component with this tag and has the GazeFocusableTag will only be considered if the distance between the GTOM source and the component is shorter than the argument part of the tag. Please note that this tag cannot be used to force the GTOM line traces to be longer than default, only exclude the primitive if the distance is longer than this argument. This is an argument tag, this means you must provide the value after the tag. Usage example: GazeFocusableMaximumDistanceTag 10000
    static FString GazeFocusableLayerTag;               //A focus manager can opt to only query a subset of focusables by using a focus layer. A primitive will this tag will only be subject to the layer supplied as the argument. This is an argument tag, this means you must provide the value after the tag. Usage example: GazeFocusableLayerTag Enemies
};

The “HasGazeFocusTag” is a bit special in that it is actually applied by the GTOM system to let you poll whether a primitive component has focus or not directly on the component. The reason why some tags are FString instead of FName is that they are so called “argument tags”. That means that to properly use them, you have to provide an argument as part of the tag. If you for example would like a particular primitive component to have a different maximum range, you can use the GazeFocusableMaximumDistanceTag tag like so: GazeFocusableMaximumDistanceTag 1000. This would cause the primitive component to only participate in GTOM if it is within 10 meters of the player.

Gaze Focusable Components

The second way to mark something as gaze focusable is to use “Gaze Focusable Components”. These can be added to any actor and will modify the default gaze focusability of every primitive component on the actor, acting as a sort of “default”. It of course offers all of the same functionality as the GTOM component tags do. Using both GTOM component tags and gaze focusable components at the same time also works, and is in fact recommended. If a primitive component is affected both by a gaze focusable component and GTOM tags, the tags will take priority, overriding the default values from the gaze focusable component.

It is worth mentioning that while attaching gaze focusable components to actors that should be focusable in some way is strictly not necessary, we strongly recommend doing it for future proofing. Some types of low level GTOM might need them to run custom code on the actor, so if you don’t put them on every gaze focusable actor, you might have to update your code in the future if you want to start using these more powerful forms of GTOM.

Gaze Focusable Widgets

UI widgets are of course also objects that a user might want to focus, but since UI elements are not actors, or primitive components, we cannot use our tag, or component to mark them for focus. As such, we support UI gaze focus using something called “Gaze focusable widgets”. These exist both in slate and in UMG, but the slate API is not fully featured yet.

The radial menu here is an example type that we provide showing how to integrate other slate widgets with gaze focusable widgets. Feel free to remix this.
Here you can see a gaze focusable widget wrapping some content (in this case a border element). This means that the border (and its children) will participate in GTOM.
Gaze focusable widgets come with built in CleanUI (feature to fade out UI elements that you are not looking at to make them less intrusive) that is configurable using these properties.

Please note that gaze focus using widget GTOM currently only works on world space widgets hosted in a WidgetComponent. The CleanUI still works in screen space however. As screen space widgets are strongly recommended against in XR though, this will probably not impact you much.

Focus Manager Layers

Focus managers can be configured to only consider subsets of scene objects. This can be very useful if you have multiple mechanics that act on different subsets of your scene actors. One example could be a game where you have two types of enemies; robots and insects. Let’s then assume you have a pistol and hacking spells. The pistol should target both the insects and the robots, while the hacking spells should only target the robots. In this situation, if you only had one focus manager that produced one focus candidate, if that focus candidate ended up being an insect, your hacking spells would most likely not work as expected in some scenarios. Since you can poll all of the focus candidates from a focus manager, you can of course just leave all gaze focusable objects in the default layer and then iterate the focused actors yourself, but it is easier to just use the focus layers since it allows for more complex subsets. You can specify what layer a primitive component is in either by using the GazeFocusableLayerTag, or using a gaze focusable component with a default layer. You can then add your layer requirements to your focus manager either via white list (only allows primitive layers that are in the list) or via black list (excludes primitive layers that you specified from focus).