Best way to mask 2D sprites in XNA?

I currently am trying to mask some sprites. Rather than explaining it in words, I've made up some example pictures:

The area to mask (in white)

Now, the red sprite that needs to be cropped.

The final result.

Now, I'm aware that in XNA you can do two things to accomplish this:

1. Use the Stencil Buffer.


2. Use a Pixel Shader.

I have tried to do a pixel shader, which essentially did this:

float4 main(float2 texCoord : TEXCOORD0) : COLOR0
{
    float4 tex = tex2D(BaseTexture, texCoord);
    float4 bitMask = tex2D(MaskTexture, texCoord);

    if (bitMask.a > 0)

    { 
        return float4(tex.r, tex.g, tex.b, tex.a);
    }
    else
    {
        return float4(0, 0, 0, 0);
    }
}

This seems to crop the images (albeit, not correct once the image starts to move), but my problem is that the images are constantly moving (they aren't static), so this cropping needs to be dynamic.

Is there a way I could alter the shader code to take into account it's position?

---

Alternatively, I've read about using the Stencil Buffer, but most of the samples seem to hinge on using a rendertarget, which I really don't want to do. (I'm already using 3 or 4 for the rest of the game, and adding another one on top of it seems overkill)

The only tutorial I've found that doesn't use Rendertargets is one from Shawn Hargreaves' blog over here. The issue with that one, though is that it's for XNA 3.1, and doesn't seem to translate well to XNA 4.0.

It seems to me that the pixel shader is the way to go, but I'm unsure of how to get the positioning correct. I believe I would have to change my onscreen coordinates (something like 500, 500) to be between 0 and 1 for the shader coordinates. My only problem is trying to work out how to correctly use the transformed coordinates.

Thanks in advance for any help!

Answer

You can use your pixel shader approach but it is incomplete.

You will also need to add some parameters to it that inform the pixel shader where you want your stencil to be located.

sampler BaseTexture : register(s0);

sampler MaskTexture : register(s1) {
    addressU = Clamp;
    addressV = Clamp;
};

//All of these variables are pixel values
//Feel free to replace with float2 variables
float MaskLocationX;
float MaskLocationY;
float MaskWidth;

float MaskHeight;
float BaseTextureLocationX;  //This is where your texture is to be drawn
float BaseTextureLocationY;  //texCoord is different, it is the current pixel
float BaseTextureWidth;
float BaseTextureHeight;

float4 main(float2 texCoord : TEXCOORD0) : COLOR0
{
    //We need to calculate where in terms of percentage to sample from the MaskTexture
    float maskPixelX = texCoord.x * BaseTextureWidth + BaseTextureLocationX;

    float maskPixelY = texCoord.y * BaseTextureHeight + BaseTextureLocationY;
    float2 maskCoord = float2((maskPixelX - MaskLocationX) / MaskWidth, (maskPixelY - MaskLocationY) / MaskHeight);
    float4 bitMask = tex2D(MaskTexture, maskCoord);

    float4 tex = tex2D(BaseTexture, texCoord);

    //It is a good idea to avoid conditional statements in a pixel shader if you can use math instead.
    return tex * (bitMask.a);
    //Alternate calculation to invert the mask, you could make this a parameter too if you wanted
    //return tex * (1.0 - bitMask.a);

}

In your C# code you pass in variables to the pixel shader prior to the SpriteBatch.Draw call like this:

maskEffect.Parameters["MaskWidth"].SetValue(800);
maskEffect.Parameters["MaskHeight"].SetValue(600);