Developing an Overlay Shader for Unity (Part 2)

Overview

In this part we address occlusion issues, and potential conflicts with other assets by decoupling the effect from the camera depth/stencil buffer. We’ll also make improvements to make it easier to highlight multiple objects at the same time.

Quick recap of what we’ve been doing so far:

  1. Acquire an 8 bit texture (without depth) to store the group ID.
  2. Bind GroupID Color + Camera Depth as render target.
  3. Render all relevant objects again, to write their group ID to the GroupID texture, while also testing against camera depth and setting stencil to 1. The group ID is provided as a shader variable.
  4. Bind Camera Color + Camera Depth as render target.
  5. Render all relevant objects, and use the stored group ID to determine outlines, while testing against camera depth and stencil. Fill color and Outline color are provided as shader variable.

Issues with this method:

  1. Another asset might rely on the camera stencil buffer, resulting in a potential conflict.
  2. During Step 3, we test against the camera depth buffer. This might fail if there are overlay objects occluding one another, because the camera depth buffer only has depth information on scene objects. In such a situation, far objects might render on top of near objects.
    Cover
    Left: Incorrect ordering, Right: expected result.

Using a separate depth/stencil buffer:

  1. Acquire an 8 bit texture (with depth/stencil) to store the group ID.
  2. Bind GroupID Color + Depth as render target.
  3. Render all relevant objects to write their group ID to the GroupID texture with ZTest and ZWrite enabled, and set Stencil to 1.
  4. Bind Camera Color + GroupID Depth Stencil as render target.
  5. Render all relevant objects again, while testing against GroupID depth and stencil, and use their group ID to determine outlines. Optionally test against camera Depth when a see through effect is not desired. Fill color and Outline color, are provided as shader variable. Since the Camera depth buffer is no longer bound, we need to do our Z tests against the _CameraDepthTexture texture.

Depth testing against “_CameraDepthTexture”

In order to test the depth of a fragment against a depth value from _CameraDepthTexture we need to bring both into the same space. This example shows how to do so in linear [0, 1] space.

Calculating linear depth for a fragment

Vertex to Fragment struct

Vertex shader

Fragment shader

Calculating linear depth for a value from_CameraDepthTexture

Fragment shader

Performing the depth test

Fragment shader

Since the calculations for both vDepth and gDepth differ, we need to use a small ZBias to avoid floating point imprecision induced Z fighting

Better multi-object support

We’ll extend OverlaySelectable to offer better access to all instances. We’ll also add the option to always highlight an object.

Keeping a list of all instances is a matter of personal preference. I like my component data and logic separate, but this is not required. You could also add the fill and outline colors directly to the OverlaySelectable component. I opted to keep a list of all group colors in OverlayRenderer instead.

Downloads

Content used