Introduction

I recently wrote an overlay effect to highlight units through walls, using the following criteria:

• Ability to see through walls
• Support alpha blending
• A thin, crisp outline
• Grouping (multiple objects share the same outline if they touch)

I thought that this was a good opportunity to share not only how it works, but also my thought process behind developing the effect. Now, if all you want to do is overlay a color over an object in Unity, you could just render the object with an unlit shader. However, in order to produce more complex effects, such as outlines, or interaction with other highlighted objects, we need to get a little more creative.

Command Buffers

First we need to determine when to render our effect. Ideally it should be rendered either right before or right after other post processing effects, depending on whether we want other effects, such as “Bloom” to be applied over our effect.
We can do so using Command Buffers, which give us full control over when to render the effect.
A Command Buffer holds a list of rendering commands to be executed by unity at a specific point in the render pipeline. For more information see Extending the Built-in Render Pipeline with Command Buffers.

Alternatively, if you are using the Post Processing v2 Package (aka PPv2), you can write a custom post processor:

This has the advantage of managing the command buffer for you, plus it provides many useful utility methods and a nice inspector for the shader variables. It does come with some caveats (primarily with TAA) which I’ll address later.

Shader Setup

I’ll keep the shader very bare bones for now. All it does is output the color red.

Drawing with Command Buffers

We need to instruct unity to render a mesh that we want to be highlighted. We do so by adding a CommandBuffer.DrawMesh command to the command buffer.

This instructs Unity to render a single mesh using the provided material (our custom shader).
However we most likely want Unity to render all meshes within a GameObject and its children, not just a single mesh.

Let’s write a small extension method to render all the meshes of a GameObject (and its children) in the correct position:

We’re now at a point where we can run our first test.
To do so I added a GameObject field to the OverlayRenderer component, and linked it with the Unity safety hat™. (this is just for testing)

Neat. But in case you happen to use the PPV2 method, and have Termporal Anti Aliasing enabled,
you will notice that the overlay image is flickering/shaking:

This is caused by TAA, which jitters the camera’s projection matrix during normal rendering. It then uses a complex algorithm to compare the current samples with samples from previous frames (hence the name “temporal”) in order to produce an anti-aliased image.
The issue in our case is that PostProcessEvent.BeforeStack and PostProcessEvent.AfterStack are rendered after TAA. Thus our hat is rendered with the jittered projection matrix, but it is never handled by the TAA algorithm, and thus remains jittered.

If you do want to use TAA in your project (if you use PPv2 you probably do), there are ways to work around this.

• You could use PostProcessEvent.BeforeTransparent to render it before TAA is appplied, however this can cause weird interactions with transparent objects.
  1	[PostProcess(typeof(OverlayEffectRenderer), PostProcessEvent.BeforeTransparent, "Overlay Effect", false)]

   
• Pass a non-jittered projection matrix to the shader. This will stop the jittering, but will also prevent the overlay effect from being anti-aliased by TAA
  1	cmd.SetGlobalMatrix(ShaderIDs._NonJitteredProjection, GL.GetGPUProjectionMatrix(context.camera.nonJitteredProjectionMatrix, true));

  1 2 3 4 5 6 7

  float4x4 _NonJitteredProjection;   float4 ComputeClipPos(float4 pos) {     float4x4 vp = mul(_NonJitteredProjection, UNITY_MATRIX_V);     return mul(vp, mul(UNITY_MATRIX_M, pos));           }

   
• Use a custom command buffer to render during CameraEvent.BeforeImageEffects (see first example), which comes after transparent object rendering, but before TAA is applied.
  This method is compatible with TAA, and works regardless of whether PPv2 is being used or not. You’ll lose the nice utility functions from PPv2. But you can always look up how they’re implemented.
  I prefer this method, I can do what I want, when I want, and it doesn’t interfere with post processing

   

  1	public CameraEvent _cameraEvent = CameraEvent.BeforeImageEffects;

Mouse Picking

Let’s use mouse picking to highlight the object at the mouse cursor, instead of selecting it in the inspector.
First, I will mark objects that can be highlighted using a component. You can also use tags or layers. The advantage of using a component is that we can attach extra information. (for now I’ll keep it empty)

These objects will also need a collider for Physics.Raycast to work.
Let’s add both to scene objects that we want to display a mouse over effect for, such as the Unity safety hat™

Next, we add basic mouse picking.

Outlines

There are many different methods for creating outlines. I’ll showcase my favorite method, which produces thin, crisp, pixel perfect, camera independent outlines. This method does not rely on vertex displacement (Inverted Hull).
Instead, it uses a fragment shader for per-pixel edge detection.

In order to determine whether a pixel is on the edge of an object, we need to access its neighbors.
Let’s zoom in really close (5×5 pixels) on the edge of a highlighted object. (top left image)
The pixel to be tested is marked in white. We’ll use a 3×3 box pattern to check if any of the neighbor pixels (a total of 8 texture samples) do not belong to the current object.

Top left: original image, with the pixel to be tested marked as white.
Bottom left: Neighbor pixels belonging to the same object marked as green, and other neighbor pixels marked as red.

If any of the tested pixel does not belong to the current object, the pixel is an edge/outline pixel.
We can only test for this, if all pixels are marked ahead of time.
This means that we need to render the effect in 2 passes.

Pass 1: Marking every pixel belonging to the highlighted object

We will do so by using a new RenderTexture to store the needed information.
Since we only store an id (initially just a boolean) we don’t need a full RGBA texture. An 8 bit single channel texture is perfectly sufficient as it provides 256 distinct values.
We can request a Temporary Render Texture from Unity. This is a neat feature designed to help with situations exactly like this, and means we don’t have to worry about things like resizing the render texture when the resolution changes.

The shader code for this pass will be almost identical to what we’ve done before. Only the fragment shader has changed.

overlayIDTexture pixels are now 1.0 if they belong to the object, and 0.0 if they do not.

Pass 2: Using the collected information to highlight edge pixels

For the second pass we use the information collected during the first pass to distinguish between outline and fill.

_OutlineColor=(255,255,255,255), _FillColor=(0,157,255,101)

You can, of course, use other sampling patterns. My favorite pattern also uses 8 taps, but they’re arranged in a diamond shape rather than the box shape in the previous example. The outline will be slightly thicker, but in my opinion, provides a more pleasant result.

Left: 8 Tap box pattern. Right: 8 Tap diamond pattern.

Skinned Meshes

Currently, our overlay renderer is not compatible with skinned meshes.

This won’t work because SkinnedMeshRenderer.sharedMesh will always return the bind pose of the mesh, resulting in the following behaviour:

For rendering, use CommandBuffer.DrawRenderer instead:

As for the collider, it is best to rely on multiple primitive colliders (box, sphere, capsule) instead.
For characters, you can often times get away with using a single capsule collider. If you need more precision, you can attach additional colliders to individual bones. For example, you could approximate an arm using 2 capsule colliders.

Depth Testing

By modifying the depth test in the shader we can achieve multiple useful effects, such as looking through walls:

Left: Normal Depth Testing

Center: Ignore the depth buffer.

This allows us to “look through” objects in front of the character. It does however also look through the character model itself, resulting in a layered look. This occurs because some pixels are handled multiple times (for example, the torso behind the right arm. It’s especially obvious with the hair). A “clean” non-layered look can be achieved by making sure every pixel is only handled once.

Right: Same as center, but mark pixels using the stencil buffer, so they are only rendered once.

In the first pass (Pass 0) we simply set the stencil mask for every pixel to 1.
In the second pass, we only render when the stencil mask equals 1. We also increment it by 1 if we do so. This way each pixel can only get rendered once.

Grouping

Left: Matching group IDs. Right: Different group IDs

Currently, in the second pass, we compare the group id of a pixel to that of its neighbors. Up until now that id has always been 0 (no overlay) or 1 (overlay). If multiple objects are rendered with the same group id, they will share a single outline. If we want them separated, all we need to do is give them different group IDs. Let’s modify our first pass shader in order to write a group id to the output texture, rather than just 1 or 0. Identical

Notes

While there is much more to be said, this post is already a lot longer than I initially intended. It is also a decent point in time to provide a very basic implementation of the shader (before I’ll add more complex features). I’ll cut it off here and post a “part 2” in the near future, where I’ll go into more advanced topics / features.

Downloads

Here’s a minimal implementation of the current state of this overlay effect.
It highlights objects when hovering the mouse over them.

• OverlayRenderer.cs (Add to a single empty gameobject)
• OverlaySelectable.cs (Add to objects you want a mouse over effect for. The object needs a collider.)
• OverlayShader.shader

Content used

• Unity HDRP Template Project
• Animated Character Pack by Quaternius