I want to implement a Unity-like API for my transforms, with the ability to set forward, right, up vectors directly and change object's rotation quaternion accordingly.
I got stuck at the math behind it, though. What algorithm should we use to change the other 2 vectors?
I feel that the function should have 2 if branches and check which of the other two vectors to use to compute a cross product with the new vector, to decide which vector to use as the first and which is the second, and vice-versa in another branch.
I came up with something like this so far:
void Transform::SetForward(const glm::vec3& v) {
mForward = glm::normalize(v);
if (abs(abs(glm::dot(mForward, mUp)) - 1.0f) <= 1e-5) {
// dot is either -1 or 1 (straight up or down)
mUp = glm::cross(mRight, mForward);
mRight = glm::cross(mForward, mUp);
}
else {
mRight = glm::cross(mForward, mUp);
mUp = glm::cross(mRight, mForward);
}
FixRotationQuat();
}