3d - How to modify normal vectors with a tranformation matrix

Transformation is quite simple when it comes to Position or Color of the vertex, for Position just multiply matrix * vector and for color just don't do anything, but my question now is: How to handle normal vectors? If my thoughts are correct I would just need the rotation part of the matrix and apply it to the normal vectors, but how can I do that? Or is it better not to use a matrix for that, but instead allow only direct transformations, one at a time, so I can handle them differently for different vertex attributes?

Answer

Simply multiply the normal vector with the inverse transpose of the 3x3 sub matrix used to transform points.

Normal_transformed = Inverse ( transpose ( mat3 ) ) * Normal;

The rational behind this is that Normal vectors has the property of being perpendicular to the surface so you need the inverse transpose to retain that property. You need the 3x3 matrix and not the full 4x4 matrix is that normals are direction and shouldn't be translated. Alternatively, you can set the normal vector w component to 0, but normals are usually vec3.

Note that if your matrix only contains rotation, the inverse transpose will result in the same original matrix so you don't need to do anything.

Normal_transformed = mat3 * Normal; // In case mat3 is only rotation.

More on the topic here.