graphics programming - How do I create a wide-angle / fisheye lens with HLSL?

What are the concepts that need to be implemented in order to achieve the effect of a wide angle lens of varying extremities?

Pseudocode and specific explanation referring to the various stages of the content pipeline, as well as what information needs to be passed in from the source code to HLSL would be very useful.

Also, what are the differences between implementing a wide-angle lens, and a fisheye?

Answer

A wide-angle lens should not behave differently than other regular lens models. They just have a larger FOV (in the D3DXMatrixPerspectiveFovLH sense -- I'm assuming you use DirectX), or larger left/right and bottom/top values (in the OpenGL glFrustum sense).

I believe the really interesting part lies in modeling the fisheye lens. There's Fisheye Quake that you can study, it comes with source.

The true fisheye projection

The projection of a fisheye lens, however, is highly non-linear. In the more common (to my knowledge, which is limited to surveillance cameras) kind of lens, a point M in space is projected onto the surface of a unit hemisphere, then that surface undergoes a parallel projection onto the unit disc:

           M
             x                 M: world position
              \                M': projection of M on the unit hemisphere
               \  ______       M": projection of M' on the unit disc (= the screen)
             M'_x'      `-.
             ,' |\         `.
            /   | \          \
           /    |  \          \
          |     |   \          |

__________|_____|____\_________|_________
                M"    O        1

There are other fisheye mappings that may give more interesting effects. It's up to you.

I can see two ways to implement the fisheye effect in HLSL.

Method 1: perform the projection on the vertex shader

Advantage: almost nothing needs to be changed in the code. The fragment shader is extremely simple. Rather than:

...
float4 screenPoint = mul(worldPoint, worldViewProjMatrix);
...

You do something like this (can probably be simplified a lot):

...
// This is optional, but it computes the z coordinate we will
// need later for Z sorting.
float4 out_Point = mul(in_Point, worldViewProjMatrix);

// This retrieves the world position so that we can project on
// the hemisphere. Normalise this vector and you get M'
float4 tmpPoint = mul(in_Point, worldViewMatrix);


// This computes the xy coordinates of M", which happen to
// be the same as M'.
out_Point.xy = tmpPoint.xy / length(tmpPoint.xyz);
...

Drawbacks: since the whole rendering pipeline was thought for linear transformations, the resulting projection is exact for vertices, but all the varyings will be wrong, as well as texture coordinates, and triangles will still appear as triangles even though they should appear distorted.

Workarounds: it could be acceptable to get a better approximation by sending a refined geometry to the GPU, with more triangle subdivisions. This might also be performed in a geometry shader, but since this step happens after the vertex shader, the geometry shader would be quite complex because it would have to perform its own additional projections.

Method 2: perform the projection on the fragment shader

Another method would be to render the scene using a wide angle projection, then distort the image to achieve a fisheye effect using a fullscreen fragment shader.

If point M has coordinates (x,y) in the fisheye screen, it means it had coordinates (x,y,z) on the hemisphere surface, with z = sqrt(1-x*x-y*y). Which means it had coordinates (ax,ay) in our scene rendered with a FOV of theta such that a = 1/(z*tan(theta/2)). (Not 100% sure about my maths here, I will check again tonight).

The fragment shader would therefore be something like this:

void main(in float4 in_Point : POSITION,
          uniform float u_Theta,
          uniform sampler2D u_RenderBuffer,
          out float4 out_FragColor : COLOR)
{
    z = sqrt(1.0 - in_Point.x * in_Point.x - in_Point.y * in_Point.y);
    float a = 1.0 / (z * tan(u_Theta * 0.5));
    out_FragColor = tex2D(u_RenderBuffer, (in_Point.xy - 0.5) * 2.0 * a);

}

Advantage: you get a perfect projection with no distortions apart from those due to the pixel accuracy.

Drawback: you cannot physically view the whole scene, since the FOV cannot reach 180 degrees. Also, the larger the FOV, the worse the precision in the center of the image... which is precisely where you want maximum precision.

Workarounds: the loss of precision can be improved by performing several rendering passes, for instance 5, and do the projection in the manner of a cube map. Another very simple workaround is to simply crop the final image to the desired FOV -- even if the lens itself has a 180-degree FOV, you may wish to render only a part of it. This is called "full-frame" fisheye (which is kinda ironic, since it gives the impression that you get the "full" something while it actually crops the image).

(Note: if you found this useful but not clear enough, please tell me, I feel like writing a more detailed article about this).