unity - Alpha Blending: use the pixel with the smallest alpha

I'm writing a transparency shader where I need any intersecting textures to, wherever they overlap, take the texture pixel with the smallest alpha value and render only that, discarding the other pixel(s).

Turn this:

Into this:

In pseudocode, something along the lines of…

if ( newPixel.alpha < currentPixelInFrameBuffer.alpha )
    currentPixelInFrameBuffer = newPixel;

I can't seem to find if there is a native alpha blending command that will do this. There exists DstAlpha, which is an alpha blending property that compares against the current fragment's frame buffer source alpha value, but I obviously cannot use that in the following manner…

AlphaTest Less DstAlpha

I've found there's a BlendOp Min, which is supposed to "Use the smaller of source and destination." but I can't seem to get that to work and I can't find documentation on what it means by 'smaller'.

Answer

To get the RGB values of your scene influenced by the minimum of several alpha values from your fog, the most straightforward way to use a render texture. This requires Unity Pro.

[Edit: Unity 5 now allows the use of render textures in the free version as well. Yay! :D ]

(Some alternatives are provided below if this isn't an option)

Set up a second camera, with Culling Mask set so that it sees only your fog hole quads. Set its Background to the colour you want your fog to be, and set its Depth so that it renders before the rest of your scene.

Then place a quad with this texture in front of your scene, so that it's visible to your main camera, and it should render on top as expected.

Set your fog hole shader to use:

BlendOp Min
Blend One One

and output all-white for the colour channel. (Either by using a .tga for your fog hole texture that's all white on the colour planes, or via the shader). This ensures the holes will never modify the fog colour, only its alpha.

Alternatively, you can make the RenderTexture in code with only a single channel, and apply any colour adjustment you want when you blend the finished fog into your scene. The advantage of rendering it as a full-colour map is mainly Editor convenience, and the ability to render arbitrary content into it like roiling clouds for the holes to later punch-out.

The reason compositing directly to your frame buffer without an intermediate texture won't work is that, as soon as that first quad is drawn, the RGB values of your scene in the dark region have already been modified. A quad drawn over it later with a lighter patch can't undo that change (at least not without dividing by DstAlpha, which is not one of the available operations).

If you're not able to use RenderTextures, your next best bets are:

1. changing your visual style so the fog isn't one even shade. That way you can just stamp down dark fog quads everywhere outside the main view, rather than punching holes inside it.



2. using the Mesh class to construct a one-layer piece of geometry that can be rendered for your fog without the same texel being blended over multiple times. One straightforward way to do this is a grid with vertex colours indicating visible/obscured cells, and a fragment shader that uses a noise texture to bias this value to keep the edge rough & feathery.

Edit: here's an example of suggestion #2, quickly mocked-up with an 8x8 vertex grid blended over a happy face reference texture.