Developing an Overlay Shader for Unity (Part 1)



I recently wrote an overlay effect to highlight units through walls, using the following criteria:

  • Ability to see through walls
  • Support alpha blending
  • A thin, crisp outline
  • Grouping (multiple objects share the same outline if they touch)

I thought that this was a good opportunity to share not only how it works, but also my thought process behind developing the effect. Now, if all you want to do is overlay a color over an object in Unity, you could just render the object with an unlit shader. However, in order to produce more complex effects, such as outlines, or interaction with other highlighted objects, we need to get a little more creative.

Command Buffers

First we need to determine when to render our effect. Ideally it should be rendered either right before or right after other post processing effects, depending on whether we want other effects, such as “Bloom” to be applied over our effect.
We can do so using Command Buffers, which give us full control over when to render the effect.
A Command Buffer holds a list of rendering commands to be executed by unity at a specific point in the render pipeline. For more information see Extending the Built-in Render Pipeline with Command Buffers.


Alternatively, if you are using the Post Processing v2 Package (aka PPv2), you can write a custom post processor:

This has the advantage of managing the command buffer for you, plus it provides many useful utility methods and a nice inspector for the shader variables. It does come with some caveats (primarily with TAA) which I’ll address later.

If you’re using the High Definition Render Pipeline (HDRP), rather than the Built-in Render Pipeline you’ll need to use a custom pass instead of command buffers.

Shader Setup

I’ll keep the shader very bare bones for now. All it does is output the color red.

Drawing with Command Buffers

We need to instruct unity to render a mesh that we want to be highlighted. We do so by adding a CommandBuffer.DrawMesh command to the command buffer.

This instructs Unity to render a single mesh using the provided material (our custom shader).
However we most likely want Unity to render all meshes within a GameObject and its children, not just a single mesh.

Let’s write a small extension method to render all the meshes of a GameObject (and its children) in the correct position:

Note that this only works for non-static GameObjects!
Static batching might otherwise merge its “meshFilter.sharedMesh” with other objects.
I’m using the static list _meshFilters to store the result of GetComponentsInChildren, instead of using the version of GetComponentsInChildren that returns an array of components. This is to prevent garbage generation, as this method will be executed every game update.

We’re now at a point where we can run our first test.
To do so I added a GameObject field to the OverlayRenderer component, and linked it with the Unity safety hat™. (this is just for testing)


Neat. But in case you happen to use the PPV2 method, and have Termporal Anti Aliasing enabled,
you will notice that the overlay image is flickering/shaking:

This is caused by TAA, which jitters the camera’s projection matrix during normal rendering. It then uses a complex algorithm to compare the current samples with samples from previous frames (hence the name “temporal”) in order to produce an anti-aliased image.
The issue in our case is that PostProcessEvent.BeforeStack and PostProcessEvent.AfterStack are rendered after TAA. Thus our hat is rendered with the jittered projection matrix, but it is never handled by the TAA algorithm, and thus remains jittered.


If you do want to use TAA in your project (if you use PPv2 you probably do), there are ways to work around this.

  • You could use PostProcessEvent.BeforeTransparent to render it before TAA is appplied, however this can cause weird interactions with transparent objects.
  • Pass a non-jittered projection matrix to the shader. This will stop the jittering, but will also prevent the overlay effect from being anti-aliased by TAA
  • Use a custom command buffer to render during CameraEvent.BeforeImageEffects (see first example), which comes after transparent object rendering, but before TAA is applied.
    This method is compatible with TAA, and works regardless of whether PPv2 is being used or not. You’ll lose the nice utility functions from PPv2. But you can always look up how they’re implemented.
    I prefer this method, I can do what I want, when I want, and it doesn’t interfere with post processing


Mouse Picking

Let’s use mouse picking to highlight the object at the mouse cursor, instead of selecting it in the inspector.
First, I will mark objects that can be highlighted using a component. You can also use tags or layers. The advantage of using a component is that we can attach extra information. (for now I’ll keep it empty)

These objects will also need a collider for Physics.Raycast to work.
Let’s add both to scene objects that we want to display a mouse over effect for, such as the Unity safety hat™


Next, we add basic mouse picking.

Note the use of LateUpdate instead of Update to ensure that the position of the object is captured after any movement. (otherwise the effect might lag 1 frame behind the actual object).


There are many different methods for creating outlines. I’ll showcase my favorite method, which produces thin, crisp, pixel perfect, camera independent outlines. This method does not rely on vertex displacement (Inverted Hull).
Instead, it uses a fragment shader for per-pixel edge detection.


In order to determine whether a pixel is on the edge of an object, we need to access its neighbors.
Let’s zoom in really close (5×5 pixels) on the edge of a highlighted object. (top left image)
The pixel to be tested is marked in white. We’ll use a 3×3 box pattern to check if any of the neighbor pixels (a total of 8 texture samples) do not belong to the current object.

In this screenshot I’ve changed the overlay color to blue (it was red in earlier examples).

Top left: original image, with the pixel to be tested marked as white.
Bottom left: Neighbor pixels belonging to the same object marked as green, and other neighbor pixels marked as red.


If any of the tested pixel does not belong to the current object, the pixel is an edge/outline pixel.
We can only test for this, if all pixels are marked ahead of time.
This means that we need to render the effect in 2 passes.


Pass 1: Marking every pixel belonging to the highlighted object

We will do so by using a new RenderTexture to store the needed information.
Since we only store an id (initially just a boolean) we don’t need a full RGBA texture. An 8 bit single channel texture is perfectly sufficient as it provides 256 distinct values.
We can request a Temporary Render Texture from Unity. This is a neat feature designed to help with situations exactly like this, and means we don’t have to worry about things like resizing the render texture when the resolution changes.

The shader code for this pass will be almost identical to what we’ve done before. Only the fragment shader has changed.

overlayIDTexture pixels are now 1.0 if they belong to the object, and 0.0 if they do not.


Pass 2: Using the collected information to highlight edge pixels

For the second pass we use the information collected during the first pass to distinguish between outline and fill.

_OutlineColor=(255,255,255,255), _FillColor=(0,157,255,101)


You can, of course, use other sampling patterns. My favorite pattern also uses 8 taps, but they’re arranged in a diamond shape rather than the box shape in the previous example. The outline will be slightly thicker, but in my opinion, provides a more pleasant result.

Left: 8 Tap box pattern. Right: 8 Tap diamond pattern.

Larger patterns will result in a thicker outline. While you might be inclined to widen the outline like this, please keep in mind how many texture samples you’re taking. A 3×3 pattern requires 8 taps per pixel, which is perfectly fine. A 5×5 pattern requires 24 taps per pixel. A 7×7 pattern needs 48 taps. This quickly grows out of control. If you need a properly thick outline, I recommend looking at the Inverted Hull method instead.

Skinned Meshes

Currently, our overlay renderer is not compatible with skinned meshes.

This won’t work because SkinnedMeshRenderer.sharedMesh will always return the bind pose of the mesh, resulting in the following behaviour:

Do not use SkinnedMeshRenderer.BakeMesh to circumvent this issue.
Yes, you can use it to acquire a new mesh every update, but constantly removing and introducing new mesh colliders is a huge strain on the physics engine. Don’t do this.

For rendering, use CommandBuffer.DrawRenderer instead:


As for the collider, it is best to rely on multiple primitive colliders (box, sphere, capsule) instead.
For characters, you can often times get away with using a single capsule collider. If you need more precision, you can attach additional colliders to individual bones. For example, you could approximate an arm using 2 capsule colliders.


Depth Testing

By modifying the depth test in the shader we can achieve multiple useful effects, such as looking through walls:
Left: Normal Depth Testing

Center: Ignore the depth buffer.

This allows us to “look through” objects in front of the character. It does however also look through the character model itself, resulting in a layered look. This occurs because some pixels are handled multiple times (for example, the torso behind the right arm. It’s especially obvious with the hair). A “clean” non-layered look can be achieved by making sure every pixel is only handled once.


Right: Same as center, but mark pixels using the stencil buffer, so they are only rendered once.

In the first pass (Pass 0) we simply set the stencil mask for every pixel to 1.
In the second pass, we only render when the stencil mask equals 1. We also increment it by 1 if we do so. This way each pixel can only get rendered once.

Using the stencil buffer to achieve this effect is a neat little trick, but unfortunately it does have some limitations. If more control is needed, or another asset already uses the stencil buffer at this stage of rendering, it might be best to utilize a separate depth buffer. I will elaborate further on this in my next post.


Left: Matching group IDs. Right: Different group IDs


Currently, in the second pass, we compare the group id of a pixel to that of its neighbors. Up until now that id has always been 0 (no overlay) or 1 (overlay). If multiple objects are rendered with the same group id, they will share a single outline. If we want them separated, all we need to do is give them different group IDs. Let’s modify our first pass shader in order to write a group id to the output texture, rather than just 1 or 0. Identical


While there is much more to be said, this post is already a lot longer than I initially intended. It is also a decent point in time to provide a very basic implementation of the shader (before I’ll add more complex features). I’ll cut it off here and post a “part 2” in the near future, where I’ll go into more advanced topics / features.


Here’s a minimal implementation of the current state of this overlay effect.
It highlights objects when hovering the mouse over them.

Content used

Notify of
Newest Most Voted
Inline Feedbacks
View all comments
Simon Stix

Hey, I just wanted to tell you that your section on grouping saved me from creating a really overengineered post-processing solution!
I’m dynamically setting groups for objects on screen, so different objects can be easily distinguished by their outline.
I never really used stencil buffers before, so thanks for probably saving me days of work.


Hi i am sorry to write after such a long time but i am getting the following errors after downloading your scripts.

“Assets\MainCar\Overlay\OverlayRenderer.cs(146,28): error CS1061: ‘CommandBuffer’ does not contain a definition for ‘DrawAllMeshes’ and no accessible extension method ‘DrawAllMeshes’ accepting a first argument of type ‘CommandBuffer’ could be found (are you missing a using directive or an assembly reference?)”
On line 120,126,139,146

I hope you can help.


Hi Jeff thanks for replying…No,I missing this,but now i have created thank you. And i have included what is written above. I now have CommandBufferExtensions.cs which is the only thing throwing 4 errors now
The type or namespace name ‘CommandBuffer’
The type or namespace name ‘GameObject’
The type or namespace name ‘Material’
The type or namespace name ‘List’

I am getting better with using and understanding code but this has me pulling my hair out.
This writeup is the best explained i have found after weeks of looking but seems i still have errors. Am i just missing something obvious i should be adding in myself?

(sorry i must add its all on line 4)

Last edited 1 year ago by fizzywizzy

I can’t seem to edit or delete above post anymore.
all errors are line 5 apart from The type or namespace name ‘List’ could not be found (are you missing a using directive or an assembly reference?) which is line 4.
Is there an easier way to contact about this as i don’t think this is the right area for such posts.
Thank you