camera - 3D Ray Casting / Picking

I am not sure if I should post this link, but I feel this falls into game development just as much as it does math. I have a ray cast's far and near values and I am trying to calculate the end point of the ray so that I can either draw a line to show the ray, or render an object at the end of 3d mouse position.

Here is my post on the stack overflow math site, if there is a problem with me doing this just close the thread. Thank you. https://math.stackexchange.com/questions/15918/help-with-matrix-mathematica

Answer

I think that you don't know 3D position of the clicked point, because otherwise it would be easy :).

So if you know 3D position of the clicked point in origin basis then you can calculate far and near points like this:

nearPoint3D = cameraposition3D
farPoint3D = cameraposition3D + (normalize(clickedPoint3D - cameraposition3D) * zFar)

The problematic part is clickedPoint3D. You can compute it nicely with help of view and projection matrices. As you are using opengl i suppose that you have them. view * projection matrix converts 3D points to the 2D screen space. You can use inverted viewprojection matrix and get matrix which converts 2D screen points to the 3D space.

pseudo code and some openGl:

//define matrices

Matrix4x4 g_CameraViewMatrix;
Matrix4x4 g_CameraProjectionMatrix;

//read projection matrix
glMatrixMode(GL_PROJECTION);    
glLoadIdentity();
gluPerspective(...); //your perspective function
glGetFloatv(GL_PROJECTION_MATRIX, &g_CameraProjectionMatrix.m[0]); //returns matrix as float array

//read view matrix

glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
glLookAt(...); //you lookat function or whatever - NO translation,rotations or anything else for getting camera matrix!
glGetFloatv(GL_MODELVIEW_MATRIX, &g_CameraViewMatrix.m[0]);  

//compute 3D position of clicked point
Matrix4x4 viewProjInv = (g_LightViewMatrix * g_CameraViewMatrix).getInverse();
vector4D clickedPointOnSreen = (x_screen,y_screen,1.0f,1.0f); // Z have to be between 0 - 1 and W have to be 1
vector4D clickedPointIn3DOrigin = viewProjInv * clickedPointOnSreen; 
vector3D clickedPoint3D = clickedPointIn3DOrigin.xyz; 

Note that you can use this to compute 3D points of zNear and zFar really easily:

vector4D clickedPointOnSreen = (x_screen,y_screen,1.0f,1.0f); // will produce 3D position of zFar
vector4D clickedPointOnSreen = (x_screen,y_screen,0.0f,1.0f); // will produce 3D position of zNear. 

Isn't that algebra wonderful? :)

To learn something about projection matrix you can read this what-is-the-purpose-of-the-canonical-view-volume