☰
Navigation :
Tobii XR Devzone > Develop > Tobii XR SDK > Unity Samples >
Hand-Eye Coordination

Hand-Eye Coordination

The Hand-Eye Coordination example showcases two gaze interactions:

  1. Gaze assisted throwing
  2. Picking up objects with gaze

This example also shows a slightly advanced usage of Tobii G2OM, which will be explained in more detail later on this page.

Interaction in this example is done using a VR controller, with the Trigger button being used to pick up and throw objects.

To learn about how to design for the Hand-Eye Coordination use case or eye tracking in general, check out our design section.

Compatibility

This particular Unity Sample is compatible with the following headsets:

HMD Compatible Notes
HP Reverb G2 Omnicept Yes Getting Started
Pico Neo 3 Pro Eye Yes Getting Started (Make sure Step 6 has been addressed in the sample scene)
Pico Neo 2 Eye Yes Getting Started (Make sure Step 6 has been addressed in the sample scene)
HTC VIVE Pro Eye Yes Getting Started
Tobii HTC VIVE Devkit Yes Getting Started

Controller haptics experience can vary on different devices.

Table of Contents

Running the Sample

Open the Example_HandEyeCoordination scene found under the Samples folder after importing the sample using the Package Manager.

If you are using Universal RP, you need to upgrade the shaders in the scene.

Scene Contents

The root objects in the scene are:

  • TobiiXR has the ThrowingSceneManager component attached which contains customized TobiiXR settings, initiating the SDK and Tobii G2OM. It has the GazeGrab and ThrowAtGaze components attached, which handle gaze assisted grabbing of objects and throwing of objects respectively.
  • XR Rig has the XR camera rig as well as a controller object as a child.
  • Environment contains the objects in the scene which are non-interactable, such as the room and lighting.
  • Gaze Throw Targets contains objects which act as potential targets in the gaze assisted throwing.
  • Grabbable Ball is a ball which the user can pick up by looking at it and pressing the trigger.
  • Throwable Ball can be picked up by looking at it and pressing the trigger, and will also assist the user when throwing it at one of the Gaze Throw Targets.

The orange ball is the Throwable Ball, the blue ball is the Grabbable Ball and the blue boxes in the scene are the Gaze Throw Targets.

TobiiXR

In this scene, a custom G2OM instance is created in the ThrowingSceneManager which is located on the TobiiXR game object. This custom G2OM instance allows for dynamically switching which type of objects should be gaze focusable. There are two types of objects which are interactable with gaze:

  • The grabbable balls, which can be picked up by looking at them and pressing a button.
  • The gaze throw targets, which can be thrown at when a ball is grabbed to the hand.

The G2OM instance is instructed to only map gaze to the two grabbable balls if no ball is being held. If a ball is being held, the G2OM instance is instead instructed to map only to the targets. Doing this separation allows for better object selection, and objects which are not interactable can be filtered out before being sent into G2OM. The ThrowingSceneManager has a reference to the GazeGrab component to be able to tell if the user is currently grabbing a ball.

Visual Feedback

A HighlightAtGaze script is placed on the Grabbable Ball and the Throwable Ball to let the user know when the ball has focus. This is to let the user know that it is interactable, which in this case means that it can be picked up by pressing the Trigger button.

Grabbing with Gaze

In the scene, there are two objects, Grabbable Ball and Throwable Ball which can be grabbed by looking at the object and pressing the Trigger button. The logic for grabbing in the scene is handled by the GazeGrab component which is located on the TobiiXR game object. The GazeGrab component looks for objects of the type GazeGrabbableObject to see if any are focused when the trigger is pressed.

The GazeGrab uses TobiiXR.FocusedObjects to see if any GazeGrabbableObject is being focused. When the user focuses on a GazeGrabbableObject, the GazeGrab component stores the object for up to 100 ms, unless the user focuses on another object of the GazeGrabbableObject type. Adding this extra time allows for the target to lose focus when the user presses the trigger to grab it, which could be because the user has looked away just before pressing the trigger. The extra time added also gives more room for the G2OM instance to miss the object at the moment when the user presses the trigger.

The GazeGrabbableObject component has two values which are modifiable in the inspector; the animation time for when the object flies to the hand when grabbed, and also the animation curve of the fly animation.

Throwing with Gaze

In the scene there is also an example of gaze assisted throwing. This example only works on non moving targets.

For throwing with gaze to work, three components are required:

  • GazeThrowableObject, which is attached to the object the user should be able to throw using gaze assist.
  • GazeThrowableTarget, which is attached to objects that the user should be able to hit with the thrown object.
  • ThrowAtGaze, which will adjust the throw trajectory within the bounds set by the GazeThrowableObject to hit the GazeThrowableTarget.

The ThrowAtGaze script will take the information of the thrown object and the target and call a native plugin to do calculations for adjusting the throw. The ThrowAtGaze has the same behavior as GrabAtGaze when selecting which object which should be focused, meaning the focus lingers on the last object for up to 200 ms. When throwing, this extra linger time can help to keep focus on the correct target at the moment when the user releases the object from the hand.

The Throwable Ball has the GazeThrowableObject component attached to it, which exposes a few values in the inspector which can be changed to modify its behavior when thrown. The values exposed in the inspector define how much assistance ThrowAtGaze gives when throwing. Modifying these values allows for changing the behavior of the gaze assist from always hitting no matter what, to hitting only when it is very close but would not hit without the adjustment.

  • Lower Velocity Multiplier is the lower multiplier from the original to the adjusted throw velocity. A value of 0.5 means that the script will adjust the velocity down to half the original throw velocity.
  • Upper Velocity Multiplier is the upper multiplier from the original to the adjusted throw velocity. A value of 2 means that the script will adjust the velocity up to twice the original throw velocity.
  • Max Y Velocity Modifier is how much the Y component (the up direction) of the throw will be adjusted from the original throw. A value of 1 will adjust the y component value of maximum one unit (which will equate to 1 m/s using standard Unity scale).
  • Xz Angle Threshold Degrees is how many degrees left or right of the target the original throw is allowed to be.