opengl - How can I convert a mouse click to a ray?

I have a perspective projection. When the user clicks on the screen, I want to compute the ray between the near and far planes that projects from the mouse point, so I can do some ray intersection code with my world.

I am using my own matrix and vector and ray classes and they all work as expected.

However, when I try and convert the ray to world coordinates my far always ends up as 0,0,0 and so my ray goes from the mouse click to the centre of the object space, rather than through it. (The x and y coordinates of near and far are identical, they differ only in the z coordinates where they are negatives of each other)

GLint vp[4];
glGetIntegerv(GL_VIEWPORT,vp);
matrix_t mv, p;
glGetFloatv(GL_MODELVIEW_MATRIX,mv.f);
glGetFloatv(GL_PROJECTION_MATRIX,p.f);
const matrix_t inv = (mv*p).inverse();
const float
    unit_x = (2.0f*((float)(x-vp[0])/(vp[2]-vp[0])))-1.0f,
    unit_y = 1.0f-(2.0f*((float)(y-vp[1])/(vp[3]-vp[1])));
const vec_t near(vec_t(unit_x,unit_y,-1)*inv);

const vec_t far(vec_t(unit_x,unit_y,1)*inv);
ray = ray_t(near,far-near);

What have I got wrong? (How do you unproject the mouse-point?)

Answer

I recently had to solve this myself for a WebGL application. I've attached the complete source code, but incase it doesn't work right off the bat for you here are some debugging tips:

1. Don't debug your unproject method in your game. If possible, try to write unit-test style tests to make it easier to isolate what is going wrong.


2. Be sure to print the output rays for both the near and far clipping planes.


3. Remember that matrix math is NOT commutative. A x C != C x A. Double check your math.

Also, to reply to some comments above, you almost never want to use OpenGL's selection APIs. That helps you pick existing items, like if you were creating a menu, however it fails to perform most real-world scenarios like 3D model editing. Where you need to add geometry as a result of the click.

Here is my implementation. There is nothing magic going on here. Just JavaScript and Google's Closure library.

gluUnProject

/**
 * Port of gluUnProject. Unprojects a 2D screen coordinate into the model-view
 * coordinates.
 * @param {Number} winX The window point for the x value.
 * @param {Number} winY The window point for the y value.
 * @param {Number} winZ The window point for the z value. This should range

 *    between 0 and 1. 0 meaning the near clipping plane and 1 for the far.
 * @param {goog.math.Matrix} modelViewMatrix The model-view matrix.
 * @param {goog.math.Matrix} projectMatrix The projection matrix.
 * @param {Array.} view the viewport coordinate array.
 * @param {Array.} objPos the model point result.
 * @return {Boolean} Whether or not the unprojection was successful.
 */
octorok.math.Matrix.gluUnProject = function(winX, winY, winZ,
                        modelViewMatrix, projectionMatrix,
                        viewPort, objPos) {

  // Compute the inverse of the perspective x model-view matrix.
  /** @type {goog.math.Matrix} */
  var transformMatrix =
    projectionMatrix.multiply(modelViewMatrix).getInverse();

  // Transformation of normalized coordinates (-1 to 1).
  /** @type {Array.} */
  var inVector = [
    (winX - viewPort[0]) / viewPort[2] * 2.0 - 1.0,
    (winY - viewPort[1]) / viewPort[3] * 2.0 - 1.0,

    2.0 * winZ - 1.0,
    1.0 ];

  // Now transform that vector into object coordinates.
  /** @type {goog.math.Matrix} */
  // Flip 1x4 to 4x1. (Alternately use different matrix ctor.
  var inMatrix = new goog.math.Matrix([ inVector ]).getTranspose();
  /** @type {goog.math.Matrix} */
  var resultMtx = transformMatrix.multiply(inMatrix);
  /** @type {Array.} */

  var resultArr = [
    resultMtx.getValueAt(0, 0),
    resultMtx.getValueAt(1, 0),
    resultMtx.getValueAt(2, 0),
    resultMtx.getValueAt(3, 0) ];

  if (resultArr[3] == 0.0) {
    return false;
  }


  // Invert to normalize x, y, and z values.
  resultArr[3] = 1.0 / resultArr[3];

  objPos[0] = resultArr[0] * resultArr[3];
  objPos[1] = resultArr[1] * resultArr[3];
  objPos[2] = resultArr[2] * resultArr[3];

  return true;
};

Usage

  this.sys.event_mouseClicked = function(event) {
    // Relative x and y are computed via magic by SystemModule.
    // Should range from 0 .. viewport width/height.
    var winX = event.relativeX;
    var winY = event.relativeY;
    window.console.log('Camera at [' + me.camera.position_ + ']');
    window.console.log('Clicked [' + winX + ', ' + winY + ']');

    // viewportOriginX, viewportOriginY, viewportWidth, viewportHeight

    var viewPort = [0, 0, event.viewPortWidth, event.viewPortHeight];
    var objPos = [];  // out parameter.

    // The camera's model-view matrix is the result of gluLookAt.
    var modelViewMatrix = me.camera.getCameraMatrix();
    // The perspective matrix is the result of gluPerspective.
    var perspectiveMatrix = pMatrix.get();

    // Ray start
    var result1 = octorok.math.Matrix.gluUnProject(

      winX, winY, 0.0,
      modelViewMatrix, perspectiveMatrix,
      viewPort, objPos);
    window.console.log('Seg start: ' + objPos + ' (result:' + result1 + ')');

    // Ray end
    var result2 = octorok.math.Matrix.gluUnProject(
      winX, winY, 1.0,
      modelViewMatrix, perspectiveMatrix,
      viewPort, objPos);

    window.console.log('Seg end: ' + objPos + ' (result:' + result2 + ')');
  };
};