Principles of Foveation
With foveation we describe all technologies that exploit the physiological properties of our vision system for better system performance with the help of an eye tracker.
As outlined in the section of the eye, the namesake of foveation technologies is taking into account the position of the eye’s foveal region, the center of high visual acuity and color vision:
Foveated rendering is an example of foveation, with the aim to capitalize on the limitations of the human visual system to reduce the processing required to generate computer imagery.
Generally speaking, for foveation to be useful, three requirements must be met:
- While interacting with a display an eye tracker measures the user’s gaze with sufficient accuracy and latency.
- The application can react to these measurements in real-time, e.g., by saving processing or bandwidth.
- These changes become imperceptible, or at least acceptable to the user, as they sink below the noise floor of all other artifacts occurring in the scene.
Once they are fulfilled foveation can be considered successful.
In practice, there is a distinct interplay between these factors for each foveation use case, and the engineering challenge is to balance them in appropriate ways to maximize savings, while staying within an acceptable perceptual threshold. Expanding on the points above, this means:
Real Time Eye Tracking
For foveation to work, the display must be updated in such a way as to stay below the user’s perception threshold. At the foundation of this lies real-time eye tracking able to provide a signal of the user’s eye movements with:
- sufficient accuracy, so that the foveated area can be kept small, and optimizations give good yield
- minimal latency, which also affects the yield, and generally allows the application to react at all, and
- high enough frequency to pick up eye movements in time from time of onset.
There are more factors to consider over a wider population and application range, but these are of critical importance in any use case.
For foveation to be generally meaningful, there needs to be some resource that can be saved. As of today, there are two large domains, rendering and transports.
- In foveated rendering the resource to save is GPU processing, which can in turn translate to faster rendering, better graphics, or reduced power consumption. Foveated rendering exploits that some parts of the real time rendering pipeline can be done more efficiently if it is known their results cannot be sufficiently perceived by the user.
- Foveated transports saves network or transmission bandwidth. In these use cases, a rendered image can be sent with more aggressive compression in areas that lie outside of the user’s foveal region, to allow for higher resolutions, reduced bandwidth utilization or again lower latency.
Every part of the processing of foveated imagery introduces delays (latency) which give time for the eye to have moved before seeing the generated images. In addition, even the very best eye trackers have limits to their accuracy and mishandling of eye tracking data can add further errors. To handle this, we need to increase the size of the area that is described as ‘foveal’ by a significant angular margin. We also need to ensure that the falloff in detail is gradual and minimizes the creation of visible artifacts that make the user aware of the foveation and non-visible artifacts that can trigger saccades and discomfort.