sentence choice - "Of these two birds the male is that which is colored brighter"

As far as I understand, in American English there must stand that instead of which in the sentence

"Of these two birds the male is that which is colored brighter"

the clause being restrictive. On the other hand, "that that" is definitely not an option. Does it mean that the sentence is impossible in formal writing and must be reworded?

Answer

There's nothing wrong with that which here.

You are mistaken in your belief that that must be employed with restrictive relative clauses: both that and wh- relatives may be used in this context.

The idea of employing only that with restrictive relatives was first advanced in 1851, at a time when grammar-writers were inclined to rationalize the language. It was given wide currency by the Fowler brothers' The King's English, which argued that "[I]f we are to be at the expense of maintaining two different relatives, we may as well give each of them definite work to do", and by the elder Fowler's even more influential Modern English Usage. It was subsequently adopted by some fairly reputable style guides.

But it is not a rule in any register, formal or informal. Some people follow it, others do not; and even those who follow it acknowledge many situations where it not only may be suspended but must be. Fowler himself acknowledged that "[I]t would be idle to pretend that it is the practice either of most or of the best writers."