projection - Using gluUnProject to transform mouse position to world coordinates (LWJGL)?

I have a little LWJGL application and I have been trying to figure out how to successfully use the GLU.gluUnProject function in order to transform a given point on the screen (mouse position) to world coordinates. I found several examples on Google and really tried hard to get it to work. I do manage to get a result vector but the values it gives me make no sense.

I initialize my scene using GL11.glFrustum(...) as follows:

public void initialize()
{
GL11.glMatrixMode(GL11.GL_PROJECTION);
GL11.glLoadIdentity();

GL11.glFrustum(-0.5, 0.5, -0.5, 0.5, 1.5, 6.5);
// left, right, bottom, top, zNear, zFar

GL11.glMatrixMode(GL11.GL_MODELVIEW);
GL11.glLoadIdentity();

GL11.glTexEnvi(GL11.GL_TEXTURE_ENV, GL11.GL_TEXTURE_ENV_MODE, GL11.GL_REPLACE);
}

The window has a size of 800 x 600. Here is a screenshot:

Now to find a corresponding point in the game world for a given mouse position on the screen I use the following method:

public void calcWorldCoordinates(float mouseX, float mouseY) 
// mouseX is in the range of [0..800] and mouseY in [0..600]
{
FloatBuffer model = BufferUtils.createFloatBuffer(16);
FloatBuffer projection = BufferUtils.createFloatBuffer(16);
IntBuffer viewport = BufferUtils.createIntBuffer(16);

GL11.glGetFloat(GL11.GL_MODELVIEW_MATRIX, model);

GL11.glGetFloat(GL11.GL_PROJECTION_MATRIX, projection);
GL11.glGetInteger(GL11.GL_VIEWPORT, viewport);

float viewportHeight = viewport.get(3); // = 600.0f
float y = viewportHeight - mouseY;

FloatBuffer posNear = BufferUtils.createFloatBuffer(16);
FloatBuffer posFar = BufferUtils.createFloatBuffer(16);

GLU.gluUnProject(mouseX, y, -1.5f, model, projection, viewport, posNear);

GLU.gluUnProject(mouseX, y, -6.5f, model, projection, viewport, posFar);

// ...
}

For mouseX = 300, mouseY = 400 I receive:

posNear = [x=-0.0580, y=0.0774, z=-0.6964]
posFar = [x=-0.0208, y=0.0278, z=-0.6964]

None of the three resulting coordinates makes sense to me. As far as I can tell x should be somewhere around 240 (each tile in the scene is of size 120 x 80 pixels). It should definately be positive though and not -0.0580 or -0.0208.

I also looked at the model and projection matrix as well as the viewport size. Those are:

model:
1.0, 0.0, 0.0, 0.0
0.0, 1.0, 0.0, 0.0
0.0, 0.0, 1.0, 0.0
0.0, 0.0, 0.0, 1.0

projection:
3.0, 0.0, 0.0, 0.0
0.0, 3.0, 0.0, 0.0

0.0, 0.0, -1.6, -1.0
0.0, 0.0, -3.9, 0.0

viewport: 0, 0, 800, 600

I rewrote the code several times using different implementations of other people but could not get it to work. Maybe somebody could give me a hint. What I am doing wrong?

Answer

gluUnProject(float winx, float winy, float winz, FloatBuffer modelMatrix,
    FloatBuffer projMatrix, IntBuffer viewport, FloatBuffer obj_pos)

winx ,winy and winz are the screen coordinates which you want to transform back to world coordinates.

winx and winy are in this case the x and y coordinates of the mouse as returned from Mouse.getX() and Mouse.getY(). In your example you have these lines:

float viewportHeight = viewport.get(3);
float y = viewportHeight - mouseY;

The reason you may see this in tutorials is that in Windows the origin used for mouse coordinates is the top left corner of the display while openGL uses the bottom left corner, so these lines are needed to transform the Windows mouse coordinates to openGL coordinates. However LWJGL already does this for us as the origin used for Mouse.getY() and Mouse.getX() is the bottom left corner, so this is not needed.

winz is the depth into the screen of the point that you want to transform back to world coordinates. The near clip plane is at z=0 and the far clip plane is at z=1. If you want the depth of the object that you clicked on, (in order to find its world coordinates), you can use glReadPixels like this:

FloatBuffer z = BufferUtils.createFloatBuffer(1);
glReadPixels(x, y, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, z);

which reads the depth of the pixel at x, y (Mouse.getX(), Mouse.getY() in this case) from the depth buffer and stores it in z. Note that the time at which you call this does make a difference, for example if you clear your buffer then check for input and call this function and then render it will not work because your buffer is empty. This can actually be useful because it allows you to render the clickable objects in your world first, then check for input, followed by rendering things you would like to be able to click through, such as a semi-transparent chat interface.

Another option which may be useful is to call gluUnProject twice with different z values to retrieve the vector from your mouse into the screen by subtracting the two points obtained.

modelMatrix, projMatrix, and viewport are the modelview matrix, projection matrix, and viewport of your world. The way you are retrieving them is correct, and may work for you for now, however if you change any of these after drawing your world, for example to render a gui over part of the display, then it may fail. To prevent this you can move your calls to glGetFloat/Integer to directly after you setup your world matrices and viewport.

obj_pos, the last parameter, is where the coordinates of the transformed point are stored if gluUnProject does not fail (which is indicated by a return value of true). In your code you create this float buffer with a capacity of 16, although it only actually needs to have a capacity of 3.