unity - Why is Raycast hitting masked layer?

Many questions have been posted on this topic but I'm facing a weird behavior which I could not find an explaination for.

I perform a raycast giving it a LayerMask parameter which is defined as public variable and set in the inspector with thick on layer "Wall" and "Enemy".

At first glance it appeared to work fine because raycast rightly stops on GameObjects with that layer.

But then I saw that it stopped also on GameObjects with default layer, like if the LayerMask was ignored. Is there something I'm missing?

According to this answer I set up everything correctly and I expected it to work fine.

Looking at the inspector at Runtime I can clearly see that my mask is set only to "Wall" and "Enemy", and that the GameObject the collider is attached to has default layer set. I'm then stating that the mask the way I'm using it does not exclude other layers. The only way I found to have this working is by setting that GameObject layer to Ignore Raycast but in this case the LayerMask is not necessary.

I'm calling the function this way:

RaycastHit2D inSight = Physics2D.Raycast(start, end, lm); 

Maybe is there something different using Physics2D?
I would really enjoy touse LayerMasks instead of giving specifically Ignore Raycast to whatever I want to ignore. Any clue?

Answer

I've had this happen with the 3D raycast in the past, and it seems to be because the raycast functions have so darned many optional parameters, and LayerMasks aren't picky about what type they're treated as.

From the docs:

public static RaycastHit2D Raycast(
       Vector2 origin, 
       Vector2 direction, 
       float distance = Mathf.Infinity, 
       int layerMask = DefaultRaycastLayers, 
       float minDepth = -Mathf.Infinity, 
       float maxDepth = Mathf.Infinity
);

layerMask is the fourth parameter. So if you provide only three parameters, Unity wants to interpret them as origin, direction, and distance.

It might seem like providing an input of type LayerMask should make your intent unambiguous, or at least generate a compile-time error, but LayerMask can implicitly convert to int, and int can implicitly convert to float.

So when you put a LayerMask in the third argument, the compiler tries to convert it to a float, succeeds, and so silently re-interprets your mask as a distance. The fourth parameter layerMask is then unspecified, so it defaults to DefaultRaycastLayers, basically ignoring the layers you chose in the Inspector.

You can fix this by providing a dummy distance value (say, infinity, if you don't have any more conservative upper bound) so your layer mask selections end up in the fourth argument where Unity expects it:

RaycastHit2D inSight = Physics2D.Raycast(start, end, float.PositiveInfinity, lm);