relative pronouns - When can we interchange “who/which" and "that”?

So far I thought the words who and that are usually interchangeable:

1. 
   

   There is the man who saw me.

   
   
   


2. 
   

   There is the man that saw me.

   
   

Both sentences seem fine, who can be replaced with that. But in the following examples:

1. 
   
   

   This is the man to whom I spoke.

   
   


2. 
   

   This is the man to that I spoke.

   
   

The #2 sounds odd, which is why I don’t think it might be correct.

The same observation for the words which and that, while

1. 
   

   This is the ball which we used today.

   
   


2. 
   

   This is the ball that we used today.

   
   

Seem to be in order and which can be replaced with that. But in the following examples the #2 sounds odd again.

1. 
   

   This is the ball with which I scored.

   
   


2. 
   

   This is the ball with that I scored.

   
   

I wonder if there’s any rule regarding the interchangeability of the above mentioned words.

Answer

Your observations are pretty precise.

In Standard English (whatever that is), relative that is not used a) to head non-restrictive ('supplementary') relative clauses or b) as the object of an immediately preceding ('pied-piped') preposition. Only wh- forms (who/whom, which) are used in these contexts.

a) That is John, {whom/^∗that} I interviewed yesterday.
b) This is the ball with {which/^∗that} I scored.

However, that may head a clause in which it acts as object of a 'stranded' preposition:

^okThis is the ball that I scored with.

In colloquial English rule a) is relaxed, because in improvised speech the distinction between restrictive and non-restrictive clauses is not so strongly marked as it is in composed speech—a restrictive clause often occurs to a speaker 'after the fact', so it is pragmatically non-restrictive even when it is semantically restrictive. But rule b) is almost universally observed in all registers.

---

^∗ marks an expression as unacceptable