Overview of how to design games with eye tracking and description of features we have explored

Creating a game experience with eye tracking can be done in many ways. Learn from reading how eye tracking relates to elements of a game, our thoughts on immersion and how eye tracking can break the 4th wall of gaming. And get insights from our comprehensive guide on the features we have explored.

Design Overview

Games can be viewed as engines of experiences, an artifact that enables real memorable gameplay experience. Regardless of your ambition, our understanding of the design games with eye tracking need to address a central challenge, to make eye tracking meaningful in a game context.

When you decide on using eye tracking in games it means that you will create game mechanics that make use the gaze signal. It could to control parts of the game with your eyes or to make the game aware of your current visual attention. The design goal to enable an engaging and memorable play experience with eye tracking is not without challenges. A good starting point is to understand what makes a game and how this structure is influenced by eye tracking.

Elements that form a game

Scholars and designers have proposed elements and structures of a game for different reasons. Scholars are suggesting for instance fundamental components that embrace a wide range of aspects to further our understanding of games. Designers have taken on a more generative purpose by formalizing games into structures that support and aid construction of new games.

With the approach to understand the relation between eye tracking and game design we need to understand what paradigmatic change a technology has on games. Jesse Schell (2008) proposes a view of the elements that forms a game. In this view the elements are aesthetics, story, mechanics and technology. No element is more important than the other, but they differ in how visible they are to the player.

• Mechanics are the rules, procedures and goals of the game. It is what makes games different to other linear entertainment experiences such as watching a movie or reading a book. Mechanics usually includes basic interaction loop but also crafting out the game as system where rules, goals and objectives forms a complete interactive engine.
• Story is the sequence of events that unfolds in the game. The power of games is that they enable the player to merge game actions to a dynamic sequence of events that either is linear in relation to a story, unforeseen as in emergent narrative elements or branched into multiple and possible story lines.
• Aesthetics are the style of the game, how it sounds, looks, feels and the tone. A player experiences the game by perceiving and interacting through the game’s aesthetics. A game can adopt a cartoonish style, borrow a theme from a popular franchise or strive to be ultra-realistic.
• Technology refers to hardware, software and components used to make the game. Regardless on views of what importance technology has on the game experience, it impacts the game and facilitate gameplay. AI let the NPC’s behave realistic, controllers enables a smooth and intuitive interaction and real time raytracing will let you perceive objects in the game with amazing qualities.

All the elements relate to each other. From every element is a link to another element to denote that they can and should support, reinforce, amplify and justify each other. When designing game mechanics you should choose a technology to support and help realize the mechanics and vice versa. Similarly, a story will guide design of mechanics to strengthen the narrative elements of the game, for instance by including exploring, combat and leveling mechanics to a journeyman adventure game. Setting an aesthetic tone can be reinforced by suitable mechanics, story, and technology to produce an outstanding experience. When for instance working with a fantasy game that includes wizardry, the game mechanics most likely should enable perceiving or possessing magical powers.

Shell comments that people find it hard to believe that all elements are of equal importance. But as a game designer you need to put equal amount of attention to all elements and sort out how they depend on each other. In the end, an aligned and tight relation between elements will mean they reinforce and support the production of a significant game experience.

But then, how should we acknowledge eye tracking? What influence does this technology have on the elements of a game?

How should eye tracking be incorporated?

In the tetrad view of game elements eye tracking is a technology change. But as argued by Schell, this means that we need to understand how technology influences aesthetic, story and mechanics.

Our experience with integrating eye tracking in games has generally involved adding eye tracking to already designed games experiences, i.e. we have infused eye tracking to a game that did not involve eye tracking when the game was originally created. This has naturally produced a separate challenge as we had limited opportunity to change story, aesthetics and mechanics without rewriting them altogether. Find out more on our list of the explored features.

Assassin’s Creed Origins. A game enhanced with eye tracking features.

Starting from a clean slate, a new game with eye tracking opens a whole range of design opportunities. Eye tracking allows the user’s gaze to be part of the game interaction, to make the user part of the game as in “breaking the 4th wall” and an extended agency.

The 4^th wall is a concept from performance theater and describes the imagined wall between actors and the audience. “Breaking the 4^th wall” thus means any form of violation to this imagined wall by for instance actors being aware and recognizing the audience in the performance. In video games, given players’ continuous interaction, this becomes a bit more complex as players are using the UI and controlling the avatar.

Yet, a special case of breaking the 4^th wall, enabled by eye tracking, is that the characters in a game can be aware and recognize the player. Eye tracking enables that the player can communicate and have a social interaction with NPC’s that also includes attention and eyes. This reinforces the social interaction to such a degree the user is included in the narrative elements of the game. Your posture, attention and gaze behavior will be responded as-if you having a normal conversation outside the game. This technological opportunity will naturally shape other elements (story, aesthetics and mechanic) during game design.

Dying Light. Keep your eyes away from the zombies, they know if you are looking at them

Immersive fallacy

It can be argued that eye tracking adds natural interaction and immersive qualities to a game. Yet, this approach probably does more harm than good to what games can be with eye tracking, at least as it risks impede development. As Frank Lantz argued in his talk from GDC 2005 and Salen & Zimmerman write about in their design book, there is an understanding of games and what games can be that falls under the concept of immersive fallacy. Immersive fallacy is the idea that games has the ability to sensually transport the players into an illusory, simulated reality that is so complete that “…ideally the frame falls away so that the player truly believes that he or she is part of an imaginary world.“ (Salen & Zimmerman, 2005). The assumption is that players are more immersed if the game includes more realism with detailed simulations of environment, people and things to do.

Yet this is not case. Games are representations just as statues, paintings plastic models represent something from reality. They would not be better if we add more details or make them closer to reality, for instance if the plastic aircraft model would be the same size as a real aircraft. What makes a game interesting is the gap between representation and reality. That some crafted interactions defined in the gap between reality and representation nurture meaning, excitement, interest and coolness. In GTA5, which offers a player sensation of enormous freedom, it is awesome to drive your car on top of buildings but probably less exciting to go to the bank and pay for your electrical bill (if that was an option). Game may include scenes with bathrooms, but that does not mean you can use bathrooms to relieve yourself. It simply makes no sense to do that act, a player will not automatically be more immersed by just being able to eat and care for other bodily needs in a game. Players are immersed in games, but not because the gap between the real world and the game is bridged, but because a designer has created an exciting, engaging and fun game experience.

Furthermore, the player is simultaneously aware of the play world and the world outside the game. The frame does not fall apart due to immersive qualities of the game.

The same goes for eye tracking. Eye tracking’s contribution to games is not increasing immersion by itself without first sorting out how, why and when it ought to be used. Gaming is a micro-cosmos where every action falls under a different set of logic and rules than the real world.

It is what you do with your eyes in the game that matters, what gameplay elements can be affected, how the play activity is changed and what meaningful play can be enabled. Therefore, it is justified to shift reference point from reality when exploring what eye tracking can do for a game and instead turn to the game world.

To summarize, we should not design and craft game mechanics with eye tracking because it is natural and has a direct impact on immersion. Immersion can be created by having ideas of how eye tracking contributes to gameplay, supported by the story and the aesthetics, to enable a meaningful play, i.e. actions that are meaningful in relation to the rules, goals, objectives of a game.

Elite Dangerous. With the eye tracking feature, the player will get better situational awareness of the surrounding.

Feature set explored

We have explored a large set of features in many prominent games. Get insight and learn from our experience on how to integrate features in a game.