opengl - First person camera with Bullet Physics

Before integrating the Bullet Physics, the camera worked fine. But instead of using my own simple struct for transform data, I use rigid body structs provided by the physics library now.

There are two issue with the first person camera at the moment. First, it is stuck in the ground and when I try to move it, it slips back to it's position. Second, When I get the camera out of the ground, which sometimes happens, The applied rotation isn't in the way I move the mouse. When I move the mouse straight, the camera is moved in circles.

This is how the rigid body it set up.

float mass = 1.0f;
btCollisionShape* shape = new btBoxShape(btVector3(0.5f, 0.5f, 0.5f));
btDefaultMotionState* state = new btDefaultMotionState(btTransform(btQuaternion(0, 0, 0, mass), btVector3(0, 0, 0)));
btVector3 inertia;
shape->calculateLocalInertia(mass, inertia);
Body = new btRigidBody(mass, state, shape, inertia);

This is how I get and set rotations and positions of rigid bodies.

vec3 Position()

{
    btVector3 pos = Body->getWorldTransform().getOrigin();
    return vec3(pos.getX(), pos.getY(), pos.getZ());
}
void Position(vec3 Position)
{
    btTransform transform = Body->getCenterOfMassTransform();
    transform.setOrigin(btVector3(Position.x, Position.y, Position.z));
    Body->setCenterOfMassTransform(transform);
}

vec3 Rotation()
{
    vec3 rot;
    Body->getWorldTransform().getBasis().getEulerZYX(rot.z, rot.y, rot.x);
    return rot;
}
void Rotation(vec3 Rotation)
{
    btTransform transform = Shape->getCenterOfMassTransform();
    transform.setRotation(btQuaternion(Rotation.y, Rotation.x, Rotation.z));

    Shape->setCenterOfMassTransform(transform);
}

And this is how I calculate the view matrix of the camera.

const float pi = glm::pi();
if      (Rotation().x < -pi) Rotation(vec3(Rotation().x + pi*2, Rotation().y, Rotation().z));
else if (Rotation().x >  pi) Rotation(vec3(Rotation().x - pi*2, Rotation().y, Rotation().z));

const float margin = 0.2f;
if      (Rotation().y < -pi/2+margin) Rotation(vec3(Rotation().x, -pi/2+margin, Rotation().z));

else if (Rotation().y >  pi/2-margin) Rotation(vec3(Rotation().x,  pi/2-margin, Rotation().z));

vec3 lookat(
    sinf(Rotation().x) * cosf(Rotation().y),
                         sinf(Rotation().y),
    cosf(Rotation().x) * cosf(Rotation().y)
);

cam->View = lookAt(Position(), Position() + lookat, vec3(0, 1, 0));

How can I move and rotate a rigid body to use it as camera?

Answer

The solution was to create a character or actor class representing a rigid body in the physics simulation. An instance of that class can be attached to the camera. Both actor and camera have distinctive transforms thus position and rotation.

The camera then updates its position and yaw from the attached actor's rigid body. Pitch and roll aren't synchronized. The position can be shifted say 1.8 meters up so that the viewpoint is located where in reality eyes are located.

Moreover the rigid body contained in an actor is restricted to not change pitch and roll, so that the collision shape cannot fall on its side.

That way the camera can freely rotate in pitch and yaw, the actor turns around with the camera but doesn't rotate upwards and downwards. So the collision shape, which might be a cylinder or an ellipsoid, always stays upright. To move the camera, we can simply apply forces to the attached actor instance.