For Engine Developers
In this section we present some useful tips if you add foveated rendering to your game engine. Although foveated rendering can be implemented for all types of renderer, the specific approaches required vary, and we primarily focus on forward and forward+ rendering.
Table of Contents
Forward (and Forward+) Rendering
A simplified overview of Forward and Forward+ rendering, together with VRS Tiers can be found on the Essential Concepts page.
Forward rendering lends itself well to hardware approaches such as Qualcomm Adreno foveation and Tier-2 buffer-space relative Variable Rate Shading.
Software temporal foveation has also been shown to be successful but the gains are lower and implementation costs higher, than with hardware approaches. If the application being foveated has not already implemented other temporal effects using accumulation and velocity buffers, then there can be significant additional resource and processing overheads.
VRS-based Foveated Rendering
When implementing VRS-based foveation, there are some steps needed during initialization, as well as some that happen each frame.
Initialization
- Create a compatible render target (this is likely to be an MSAA target)
- Create the VRS frequency control texture:
- The required resolution of this texture can be found by dividing the render target resolution by the VRS tile size (reported by the API) and rounding up to the nearest integers.
- The texture should be at least double buffered as it will be written to and consumed on separate frames.
- Bind the VRS mapping texture to the render target.
Notes:
- For single pass stereo, a single render target and VRS mapping texture must be used. If not using single pass stereo, then it is up to the implementer to choose whether to use shared, or per-eye resources.
- Some hardware implementations and APIs may also support a frequency translation table. This will be a 1-dimensional buffer which translates indices in the frequency control texture to frequency patterns.
Each Frame
- Fetch the gaze vectors and project them to 2D coordinates within the VRS mapping texture.
- Using either a compute or pixel shader, populate the VRS frequency control texture with values that have high frequencies near the projected 2D coordinate and lower frequencies elsewhere, compare below.
- Render as normal.
Populating the VRS Frequency Control Texture
Using oval or circular patterns for the frequency control layout reduces visibility of sharp delineations between different frequency regions within the display.
Ovals have the advantage of requiring only a small number of shader constants to configure them and are cheap to process in shaders creating the mapping texture. They can also be overlapped to create more ‘zones’ with finer control of, and smoother falloff in, shader frequency:
In the frequency control texture, the horizontal and vertical shader invocation frequencies should reduce relative to the respective horizontal and vertical distances from the projected foveal point. Also, to maximize use of the available VRS frequency control blocks and for simplicity of implementation it is recommended to use a series of overlapping ovals (see diagram) to control the population of the control buffer.
Qualcomm Foveated Rendering Extension
As mentioned in the Essentials section, the Qualcomm hardware foveation is an OpenGL ES extension. It maintains overall resolution skipping processing of some pixels with their content being derived from neighboring pixels.
For its use the following concepts can be relevant:
Foveated Gain
- Controls the rate of falloff in X and Y direction.
- Values greater than 1 will cause a reduction in bin resolution toward the extends while Area preserves more of the focal region.
- The gain value for X and Y can be different making fovea elliptical (having different values for width and height)
Foveation Area
- This defines the size of fovea region in which resolution does not change and this moves with user gaze. Resolution is reduced in the periphery.
Foveation Minimum
- Defines the quality level - there are 5 levels defined for Adreno foveation = (1:1, 1:2, 1:4, 1:8 and 1:16).