neologisms - How should I use the "in-", "im-", "il-", and "ir-" prefixes?

Consider the following words:

impossible;
incorrect;
impatient;
illegal;
irregular;

The meaning of the prefixes is the same (negation the adverb), but they are still different prefixes.

How do I know which one to use? In other words, for an arbitrary adjective, how do I choose between "in-", "im-", "il-", and "ir-"?

The relevant discussion at ELU does not provide with any guideline.

Answer

Formally, it is the same prefix "in-".
However, there are four different spellings that help pronunciation.

The rule only depends on the first letter of the adjective:

L — use "il-": illegal, illiterate;

P, M, or B — use "im-": improper, immediate, imbalance;
R — use "ir-": irresponsible, irrational;
other consonants or vowels — use "in-": indifferent, innumerable, inadequate;