antialiasing - How do I implement anti-aliasing in OpenGL?

I want to do full-screen anti-aliasing in OpenGL, and dont want to use the anti-aliasing that is provided by OpenGL itself, since I am building a game and want good effects in it.

How should I proceed?

Answer

There are several alternatives to native MSAA in OpenGL. With post-processing effects, the best thing about them is that you can usually just throw in the shader to the final, unprocessed image and it does the rest. Here are three methods worth taking a look:

• 
  

  Fast Approximate Anti-Aliasing (Geeks3D) - Good in most cases. Pretty easy to apply and understand. Drawback is sharp, high contrast noise in textures gets blurred a bit. Edges as subtle as 1/4 pixels steep look dead-accurate as traditional MSAA. Any less steep than that, it loses a bit of accuracy.

  
  


• 
  
  

  Normal Filtered Anti-Aliasing (GameDev) - Haven't tested this one yet accurately, but it's the easiest to understand. In best cases it resembles 16x MSAA and in worst cases it's like 2x MSAA. It generates a temporary normal map to represent edges and relative angles. You can sample the normal map either with luma difference or color difference.

  
  


• 
  

  Morphological Anti-Aliasing (Iryoku) - been improved to SMAA - Subpixel Mophological AA. It's pretty complex at 4 passes, but achieves the best results I've seen. It creates gradients along edges as gradual as 1/100 to 1/200 pixels steep (!). Sampling can be luma-based, color-based or depth-based. Textures stay very crisp and clean. (the example is DX10 and HLSL based, would take some time to port it to GLSL accurately)

  
  

These techniques don't super-sample or multi-sample, so lines that appear less than 1 pixel in thickness will appear with gaps and not be anti-aliased correctly. This is the downside to using a non-MSAA approach. Since you're only working with a raster image at full resolution, you can't create additional information from these empty gaps.

Take notice that all of these techniques are dependent on sampling adjacent luma (brightness) or chroma (color) values. Calculating luma and optional gamma correction requires additional instructions on the AA shader, though it's pretty straightforward. You can offload this by calculating the luma in the previous shader that provides the un-retouched image, storing the luma in the alpha channel. Then in the AA shader, you will simply sample the alpha.