unity - Can't figure out what's making the player teleport back to spawn

I've been working on a multiplayer game that uses pathfinding and I'm really confused over this one weird bug. On your screen, the other player glitches between the spawn point and his current location. It seems like there's a problem with the interpolation as it doesn't do this when I comment out:

transform.position = Vector3.Lerp(transform.position, bestGuessPosition, Time.deltaTime * latencySmoothingFactor);

Does this have anything to do with the way I'm moving the player?

using UnityEngine;
using UnityEngine.AI;
using UnityEngine.Networking;

public class MinionController : NetworkBehaviour {

    public Camera cam;


    public NavMeshAgent agent;

    Vector3 velocity;
    Vector3 bestGuessPosition;

    float ourLatency;
    float latencySmoothingFactor = 10;

    // Update is called once per frame
    void Update()

    {

        if (hasAuthority == false)
        {
            bestGuessPosition = bestGuessPosition + (velocity * Time.deltaTime);
            transform.position = Vector3.Lerp(transform.position, bestGuessPosition, Time.deltaTime * latencySmoothingFactor);
            return;
        }

        if (Input.GetMouseButtonDown(0))

        {
            Ray ray = cam.ScreenPointToRay(Input.mousePosition);
            RaycastHit hit;

            if (Physics.Raycast(ray, out hit))
            {
                agent.SetDestination(hit.point);
            }
        }
    }


    [Command]
    void CmdUpdateVelocity(Vector3 v, Vector3 p)
    {
        transform.position = p;
        velocity = v;

        RpcUpdateVelocity(velocity, transform.position);
    }


    [ClientRpc]
    void RpcUpdateVelocity(Vector3 v, Vector3 p)
    {

        if (hasAuthority)
        {
            return;
        }

        velocity = v;

        bestGuessPosition = p + (velocity * (ourLatency));
    }
}

Answer

Your method for blending to the new updated position definitely looks dicey. While there are times when we want to use the pattern value = lerp(value, target, sharpness) for an exponential ease-out effect, interpolating positions in network replication is not one of them. At best, it will give us inconsistent movement speeds - fast whenever a new update comes in, slower as we approach the last estimate - resulting in a stuttery, glitchy appearance.

One common solution here is to use projective velocity blending.

Here's a quick overview of the technique from a presentation on vehicle replication in Watch_Dogs 2:

The dark green arrow P0->V0 is the trajectory of the local pawn at the time of the update. The blue arrow P'0->V'0 is the trajectory according to the update packet. The orange arrow Vb represents a bending of the velocity into agreement with the new data, and the bright green squiggle P0->Pos is the interpolated position - initially agreeing with the old trajectory and gradually correcting into agreement with the update.

To do this we need to capture a bit more information when a new snapshot comes in:

struct Snapshot {
    readonly Vector3 positionOnArrival;
    readonly Vector3 velocityOnArrival;
    readonly Vector3 reportedPosition;
    readonly Vector3 reportedVelocity;
    // Using floats for time works well for games that can go for hours.
    // Switch to doubles or other formats for games expected to run persistently.
    readonly float arrivalTime;
    readonly float blendWindow;


    public Snapshot(Vector3 currentPosition, Vector3 currentVelocity,
                   Vector3 reportedPosition, Vector3 reportedVelocity,
                   float blendWindow) {
        arrivalTime = Time.time;
        positionOnArrival = currentPosition;
        velocityOnArrival = currentVelocity;
        this.reportedPosition = reportedPosition;
        this.reportedVelocity = reportedVelocity;
        this.blendWindow = blendWindow;
    }


    Vector3 EstimatePosition(float time, out Vector3 velocity) {
        float dT = time - arrivalTime;
        float blend = Mathf.Clamp01(dT/blendWindow);

        Vector3 velocity = Vector3.Lerp(velocityOnArrival, reportedVelocity, blend);

        Vector3 position = Vector3.Lerp(
                   positionOnArrival + velocity * dT,
                   reportedPosition + reportedVelocity * dT,

                   blend);

        return position;
    }
}

Note that the blending function transitions smoothly to dead reckoning from the last received update if we don't receive a new one within the expected window, so you don't need to handle the case of overdue messages separately.

Updating the snapshot:

Snapshot currentSnapshot;
// (Initialize this on Start/spawn with your spawn position)


[ClientRpc]
void RpcUpdateVelocity(Vector3 v, Vector3 p)
{
    if (hasAuthority)
        return;

    currentSnapshot = new Snapshot(
        transform.position, velocity,
        p, v,

        updatePacketPeriod // Expected time between new position/velocity reports.
    };
}

Using a snapshot:

if (hasAuthority == false)
{
    transform.position = currentSnapshot.EstimatePosition(Time.time, out velocity);
    return;
}

With the version above, the simulation is allowed to live one network update tick in the past. (ie. the reported position/velocity is considered accurate for "now" when it arrives, even though it was recorded & transmitted one transmission interval in the past)

You could optionally extrapolate the reported position into the future by (up to) your estimated latency, to reduce this time discrepancy.

The advantage of using the report as-is and living slightly in the past is that your replica will periodically reach position & velocity states that the original object actually occupied. By extrapolating, you lose this guarantee, and your replica might hover in the vicinity of the actual trajectory without hitting it exactly, though hopefully staying closer to the current position of the original on its own machine.