transformation - How to create a custom set of coordinates based on two points in Unity?

As the title says, I am having problems finding a way to create a new set of local coordinates based on two points. Here is an example:

Consider that the two gameObjects are not one parent of another, therefore I cannot use something like transform.localPosition, furthermore I tried using transform.TransformPoint(), however the latter works only if if I want to create it with respect to a single point. I think it is a silly question, yet I am not finding a way to solve it, so any help is greatly appreciated!

Answer

Based on your comments, I understand you want the second point to be on the Y+ axis of the local coordinate system you're building, and the local Z+ axis to correspond to the world Z+ axis. We can get this like so:

Matrix4x4 GetCoordinateSystemFromPoints(
              Vector3 origin,
              Vector3 pointOnYPlusAxis)
{    
    return Matrix4x4.TRS(
             origin,
             Quaternion.LookRotation(Vector3.forward, pointOnZPlusAxis - origin),
             Vector3.One
    );

}

You can transform a point from this local space to world space with:

var worldPoint = matrix.MultiplyPoint3x4(localPoint);

And transform from world space to local space with:

Matrix4x4 inverse;
Matrix4x4.Inverse3DAffine(matrix, ref inverse);

var localPoint = inverse.MultiplyPoint3x4(localPoint); 

The downside of this is that it's somewhat more expensive to invert.

You could instead define your own CoordinateSystem struct that includes both the forward and inverse matrix, or an origin & quaternion pair, so you can transform both into and out of this coordinate system frequently without repeatedly computing a matrix inverse, something like this:

public struct CoordinateSystem {
    // 7 floats instead of 12-16 for a 3x4 or 4x4 matrix.
    public readonly Quaternion orientation;
    public readonly Vector3 origin;

    public CoordinateSystem(Vector3 origin, Vector3 pointOnYPlusAxis) {
        this.origin = origin;

        this.orientation = Quaternion.LookRotation(
                              Vector3.forward, pointOnYPlusAxis - origin);
    }

    public Vector3 TransformVector(Vector3 vector) {
        return orientation * vector;
    }    

    public Vector3 InverseTransformVector(Vector3 vector) {
        // Quaternion inverse is cheaper than a matrix inverse: just 3 negations.

        // The compiler may be able to combine this with the multiplication below.
        var inverse = Quaternion.Inverse(orientation);
        return inverse * vector;
    }

    public Vector3 TransformPoint(Vector3 point) {
        return origin + TransformVector(point);
    }

    public Vector3 InverseTransformPoint(Vector3 point) {

        return InverseTransformVector(point - origin);
    }
}