Stencil Buffered Shadows

Copyright info

All code in this tutorial is protected by the [GPL License].
All text in this tutorial is protected by the [FDL License].

Tutorial by: MrG

Mailto: biteme@telefragged.com

If you've ever enabled gl_shadows, you'll most likely have noticed how the shadows are just the model casting the shadow but flattened out on the floor. The polygons of these shadows often cross over themselves and create ugly patches where the shadows are darker and lighter.

That is what we are going to fix with this tutorial, so that every pixel that has a shadow fall on it is only darkened once. To do so, we are going to use OpenGLs stencil buffer.

The stencil buffer works in a fairly simple manner, it looks at the stencil value of a pixel, compares it to a specified value, and if the operation passes, the pixel is drawn and optionally the stencil value is updated.

Now onto some code changes. This tutorial takes place in the OpenGL renderer library, so open up the gl_ref source. Locate the file 'glw_imp.c'. Above the function 'VerifyDriver', we are going to declare a variable to hold the existance of the stencil buffer for us.

 
 
qboolean have_stencil = false; // Stencil shadows - MrG
 
 

When we initialize OpenGL, we will check whether our request for a stencil buffer was successful, and if so, set the value of that variable to true.

Now move down the file till you find the function 'GLimp_InitGL'. This function is what initializes OpenGL on the games window. We are going to change the Pixel Format Descriptor so that it requests 8 bits of stencil data for us. To do so, locate the lines that read:

 
 
32, // 32-bit z-buffer 
0, // no stencil buffer
 
 

And change them to read:

 
 
24, // 24-bit z-buffer 
8, // 8bit stencil buffer
 // Stencil Shadows - MrG
 
 

Here we have made our request for the stencil buffer. You'll notice that I also changed the 32bit z (depth) buffer request to a 24bit one. This is because a 24bit depth buffer and 8bit stencil buffer pack nicely together as a 32bit value, and is more efficient for OpenGL to work with. 24bits of depth buffer is plenty.

Now we are going to set the value of our 'have_stencil' variable that we declared earlier. Locate within the 'GLimp_InitGL' function the line that reads:

 
 
ri.Con_Printf( PRINT_ALL, "GL PFD: color(%d-bits) Z(%d-bit)\n", ( int ) 
pfd.cColorBits, ( int ) pfd.cDepthBits );
 
 

Directly underneath this line, we are going to check the Pixel Format Descriptor to see if our request for a stencil buffer was successful. If it was, we are going to set the value of our 'have_stencil' variable to true.

 
 
if (pfd.cStencilBits) have_stencil = true; // Stencil shadows - MrG
 
 

That's all that needs doing in that file. Now we're moving on to the function that draws our shadows. Open up the file 'gl_mesh.c' and locate the function 'GL_DrawAliasShadow'. Directly above this function, we are going to make a reference to our externally declared variable, 'have_stencil'. We do this by declaring the variable as per normal, except we put the 'extern' keyword infront of it, and we dont set its innitial value.

 
 
extern qboolean have_stencil; // Stencil shadows - MrG
 
 

Now that we've done that, we'll move down into the function (GL_DrawAliasShadow) itself, and start adding our stencil buffer code. Locate the line that reads:

 
 
height = -lheight + 1.0;
 
 

Now, directly underneath it, we are going to setup the stencil buffer. First we do our check to make sure we have our stencil buffer to use, afterwhich we will make use of it.

 
 
// Stencil shadows - MrG
if (have_stencil) {
 qglEnable(GL_STENCIL_TEST);
 qglStencilFunc(GL_EQUAL,1,2);
 qglStencilOp(GL_KEEP,GL_KEEP,GL_INCR);
}
// End Stencil shadows - MrG
 
 

The qglStencilFunc function is telling OpenGL that it should only draw over pixels that have a stencil value equal to 1. qglStencilOp is telling OpenGL to only update the pixels if both the stencil and depth buffer tests pass, and to increment the pixels stencil value if it does pass.

After we have drawn our shadow, we want to disable the stencil buffer. Right before the function is closed, we are going to tell OpenGL to disable use of the stencil buffer.

 
 
if (have_stencil) qglDisable(GL_STENCIL_TEST); // Stencil shadows - MrG
 
 

Now we have one last change to make. Open up 'gl_rmain.c' and locate the function named 'R_Clear'. At the very end of this function, we are going to test to see if we have our stencil buffer (by checking our variable 'have_stencil'), and if we have it, clear its contents so that each pixel has a stencil value of 1.

 
 
// Stencil shadows - MrG
if (gl_shadows->value) {
 qglClearStencil(1);
 qglClear(GL_STENCIL_BUFFER_BIT);
}
 
 

Compile and run Quake2. Bring down the console, and type 'gl_shadows 1' and hit return. Shadows have now been enabled. Start a game and walk up to an enemy (be careful not to die tho!) and look at his feet. If your graphics card has granted us our stencil buffer, the shadow that extends out should be clear, but if your graphics card has denied our request for a stencil buffer, the shadows will remain ugly.

---

Update:

Two more small fixes to the shadow code. One of these fixes the issues with the shadows on gibbs that rotate wildly on axis' that it shouldnt rotate at all on.

By default, Quake2 renders its model shadows one unit above the ground, but when you look closely you can notice the shadow isnt underneath models at all, but sticking out of them partway up. The second fix lowers them to the ground more, greatly lessening the chance of noticing the floating shadows.

For the first fix, open up 'gl_mesh.c' and locate the function 'R_DrawAliasModel'. Jump to the very end of this function, into the if statement that reads:

 
 
if (gl_shadows->value && !(currententity->flags & (RF_TRANSLUCENT | RF_WEAPONMODEL)))
 
 

Now find the line that reads:

 
 
R_RotateForEntity (e);
 
 

The R_RotateForEntity sets up OpenGL to render the model at the correct position and rotation. Unfortunately it rotates the model around where we dont want it to. Replace it with:

 
 
// Dont rotate shadows on ungodly axis' - MrG
qglTranslatef (e->origin[0], e->origin[1], e->origin[2]);
qglRotatef (e->angles[1], 0, 0, 1);
// End
 
 

Those two replacement lines of code have been taken from the R_RotateForEntity function to position the entity and rotate it on just the horizontal plane.

For the second fix, move up into the function 'GL_DrawAliasShadow' and find the line that reads:

 
 
height = -lheight + 1.0;
 
 

This is what sets the height of the shadows vertices. The problem is that it sets the height one whole unit above the ground. This can be seen clearly when ducking down and looking at where the shadows come out of the models. Change this line to read:

 
 
height = -lheight + 0.1f; // lowered shadows to ground more - MrG
 
 

Now the shadows are only 0.1 units off of the ground, which is much closer than 1, and is far less noticable than before.

Thats it for the tutorial update. Compile and run! :)

Tutorial Originally found at: