glm - How to recalculate AABB after rotation?

I'm facing a problem trying to calculate the AABB on my models after rotation is applied. The rotation is not applied on the vertex positions directly, but in a seperate rotation matrix - just multiplied with the projection matrix and set as uniform. Currently I'm calculating the min/max of the vertexes on all three axes (x, y, z) to calculate the AABB. This works well, as long as the model is unrotated.

AABB calculateAabb() {
    glm::vec3 center, extend;
    glm::vec3 min, max;
    for (size_t i = 0; i < getVertexCount(); ++i) {
      if (positions[i * 3] < min.x){
        min.x = positions[i * 3];
      }
      if (positions[i * 3 + 1] < min.y) {

        min.y = positions[i * 3 + 1];
      }
      if (positions[i * 3 + 2] < min.z) {
        min.z = positions[i * 3 + 2];
      }
      if (positions[i * 3] > max.x) {
        max.x = positions[i * 3];
      }
      if (positions[i * 3 + 1] > max.y) {
        max.y = positions[i * 3 + 1];

      }
      if (positions[i * 3 + 2] > max.z) {
        max.z = positions[i * 3 + 2];
      }
    }
    extend = (max - min) * 0.5f;
    center = min + extend;
    return AABB(center, extend);
  }

The std::vector positions[] contains all vertexes in the order x, y, z - getVertexCount() returns positions.size()/3. My rotation matrix is described by a single angle on the y axis glm::eulerAngleY(glm::radians(angle));.

My question is: is there any way to calculate the AABB for the rotated model without applying the rotation matrix to all the vertexes individually? How can I get min/max values on the axes from a rotated coordinate system?

Answer

It depends on the shape of your meshes. For correct calculation of bounding boxes, only the vertices spanning the convex bounding body of a mesh have to be considered (In other words, vertices that are going "inwards" do not have to be considered).

However, depending on how often your mesh changes its shape, how often you are rotating your meshes and the shape of your meshes, it may or may not be faster to just apply the matrix to every vertex and do it the naive way than calculating which vertices need to be considered. In particular, this does not yield any performance improvement for convex shapes.

Apart from that, there are only approximations that can be made, for example:

• assume the body is a sphere for small rotations (thereby ignoring the rotation and leave the bounding box as is)


• precalculate a simpler bounding shape and use this for the AABB calculation