Benefits and Costs
Foveated rendering is an exciting technology that can improve your application and device performance if applied correctly. In this section you can read about what benefits you can gain from it and the costs which are associated with implementing it.
Table of Contents
Faster Rendering
Rendering in XR is done via multi-stage pipelines. Depending on your rendering architecture, foveated rendering can deliver performance saving in specific pipeline parts, which in turn can translate to faster rendering.
The greatest opportunities apply to the main rasterization and lighting stages. Foveation can only accelerate rendering which has a direct relation to what is visible to the user. Opportunities for dynamic foveation during post-processing are limited, some post-processing shaders can use simplified paths when not in the foveal region and some resolution reduction techniques (including VRS) can be used, though these cannot be applied as aggressively as they are during the main rasterization stages as the reduced resolutions and qualities are more likely to create artifacts. Color look-up tables, gamma passes and similar are not suitable for any form of foveation as discrepancies in the output of these passes will result in large changes in luminance.
Off-screen targets such as shadow buffers and projected lighting cannot generally be accelerated. The same applies to other GPU activities, such as physics workload and in most cases, vertex, geometry and compute processing, which do not generally benefit from foveated rendering.
Overall processing savings are relative to the proportion of the rendering pipeline processing that can be foveated. In both absolute and relative terms, savings increase with increasing resolution and pixel/fragment shader or lighting complexity. They increase the gains achieved by reducing the per-pixel processing costs and tends to increase the relative processing load of the parts of the pipeline that can be foveated.
Based on Tobii testing of Qualcomm foveated rendering and NVIDIA Variable Rate Shading foveated rendering together with third party testing of Logarithmic Remapping deferred foveated rendering have all shown 60% or greater net processing cost reductions to the foveated portions of their respective rendering pipelines.
Power Savings
These processing savings can also translate to power savings. GPU power consumption typically relates to a combination of GPU frequency and GPU utilization, where foveation can only directly affect GPU utilization.
Application developers do not generally have direct control over GPU frequency, however for hardware manufacturers that do (or can expose the control), there are some potentially significant benefits to power consumption.
Increasing GPU frequency comes at an exponentially increasing cost in terms of power consumption. Reducing the frequency even slightly can produce significant power savings. The exponential nature of the power consumption and the power floor required to maintain operation does however mean that returns decrease with larger frequency reductions.
As a simplified example, an application which is only utilizing the GPU for 50% of each frame could potentially run at half the frequency, which would consume far less power.
Visual Quality
Foveated rendering can also allow for better visual quality of your application. Since less resources have to be spent on rendering peripheral areas, the freed capacity can be reinvested in graphical fidelity. Several scenarios are possible:
The freed GPU time can be used for more complex lighting and shaders. Any computation that primarily works on fragments or pixels can benefit here.
Increasing resolution is another alternative. With the trend to larger resolutions in XR, the foveated regions will benefit particularly. With fixed resolution displays, there may not be the option to increase the final resolution, however super-sampling (rendering at a higher resolution and then filtering and shrinking the images), directed by eye tracking can give increased detail and better anti-aliasing with the processing more effectively directed than with MSAA or non-eye-tracked super-sampling.
Dynamic foveation improves the visual quality over static foveation. While static foveation allows for some savings, they are dictated by the optical properties of the device’s lenses. Already today, the effect of static foveation is quite visible and with improved optics and larger field of views the potential savings will diminish. Dynamic eye tracking foveation, in contrast, can be almost imperceivable, hence improving the visual quality.
Foveation, by definition, introduces some artifacts in the user’s visual periphery due to the lowered rendering quality. The goal is to ensure these artifacts stay below the user’s perception threshold. Similarly to how static foveation can hide these artifacts in the device lenses’ zone of imperfection, foveated rendering can hide them in the user’s peripheral vision.
To read about these in more detail, go to Artifacts and Mitigation.
Cost
The benefits above come at the cost of an increased engineering effort on the engine level, some design considerations on the application level and potentially some additional content markup and modification.
On an engineering level, artifact control means additional filtering options and exposing methods of adding mark-up to exclude some elements from the foveation.
For content creators, handling artifacts consists primarily of marking elements for exclusion from the foveation. There can also be some conflict between some content choices and specific foveation techniques. For instance, VRS foveation can produce artifacts in large primitives which use texture or parallax mapping to generate detail. These elements may need to be broken down into smaller geometry pieces or replaced with alternatives.