android - Implementing fog of war in opengl es 2.0 game

Hi game development community, this is my first question here! ;)

I'm developing a tactics/strategy real time android game and I've been wondering for some time what's the best way to implement an efficient and somewhat nice looking fog of war to incorporate in it.

The style of fog of war i'm looking for is to darken the areas where there is no visibility at a certain moment. [not the solid black, unexplored area, kind of fog of war] Note that this has nothing to do with which units are visible to the player. This is merely a visual effect.

My experience with OpenGL or Android is not vast by any means, but I think it is sufficient for what I'm asking here.

So far I have thought in some solutions:

1. 
   

   Draw white circles to a dark background, corresponding to the units visibility, then render to a texture, and then drawing a quad with that texture with blend mode set to multiply. Will this approach be efficient? Will it take too much memory? (I don't know how to render to texture and then use the texture. Is it too messy?)

   
   


2. 
   

   Have a grid object with a vertex shader which has an array of uniforms having the coordinates of all units, and another array which has their visibility range. The number of units will very probably never be bigger then 100. The vertex shader needs to test for each considered vertex, if there is some unit which can see it. In order to do this it, will have to loop the array with the coordinates and do some calculations based on distance. The efficiency of this is inversely proportional to the looks of it. A more dense grid will result in a more beautiful fog of war... but will require a greater amount of vertexes to be checked. Is it possible to find a nice compromise or is this a bad solution from the start?

   
   

Which solution is the best? Are there better alternatives? Which ones?

Thank you for your time.

Edit

I have (kinda) tried the second option with only one unit, and the shadows were getting too pixelated.. If I increased the number of vertexes, the fps started to drop. I tried doing the tests in the fragment shader and the results were better in terms of smoothness (that was expected) but also in terms of performance (comparing to the denser grid), which got me a little surprised. But I don't know if testing each pixel against 50 units might be viable to do in the fragment shader. Probably not!

I'm gonna bring a third option that I don't know if is valid, but maybe you people can enlighten me:

• 3.Use the stencil buffer in some way to draw the invisible areas in white (this would be done by having the stencil buffer initialized in white, and then draw black circle textured quads on it corresponding to units visibility) and then use that to make a kinda "clipping mask" effect on a full black somewhat transparent quad that would darken the invisible areas.

I don't even know if this is possible, and I'm not 100% sure that every android with opengl 2.0 supports stencil buffer. I also don't know very well what is the stencil buffer or when it is used... But apparently this might be a perfect solution. Does anyone have an idea?

Answer

I managed to do it with stencil buffer.

Here's how to do it: In GLSurfaceView constructor, put:

setEGLConfigChooser(5,6,5,0,16,8);

before setting the render. This makes sure you have stencil buffer active.

Then in the draw method:

    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_STENCIL_BUFFER_BIT);

    //draw other unrelated stuff here


    GLES20.glEnable(GLES20.GL_STENCIL_TEST);
    GLES20.glStencilFunc(GLES20.GL_ALWAYS, 0x1, 0xffffffff);
    GLES20.glStencilOp(GLES20.GL_REPLACE, GLES20.GL_REPLACE, GLES20.GL_REPLACE);
    GLES20.glColorMask(false, false, false, false); 

    for(Unit u: ownUnitsToDraw)
        Drawings.drawUnitVisionMask(u); //This is a quad with a texture of a white circle

    GLES20.glColorMask(true, true, true, true);
    GLES20.glDepthMask(true);

    GLES20.glStencilFunc(GLES20.GL_NOTEQUAL, 1, 1); 
    GLES20.glStencilOp(GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_KEEP);

    Drawings.drawFog(); //This is a black quad with some transparency

    GLES20.glDisable(GLES20.GL_STENCIL_TEST);

    //draw other unrelated stuff here

So what happens here is that the black fog is masked to the zone outside the vision of the units which is exactly the intended. If we set glStencilFunc to GLES20.GL_EQUAL it would be the space around the units that would have fog, but that is not wanted.

Also, in order for this to work, the mask quad fragmentation shader needs to be something like this:

    precision mediump float;
    uniform sampler2D u_Texture;
    varying vec2 vTexCoordinate;
    uniform vec4 vColor;
    void main() {
      gl_FragColor = vColor * texture2D(u_Texture, vTexCoordinate);
      if(gl_FragColor.a < 0.5)
        discard;
    };

This will allow the pixels in the quad that are transparent (alpha < 0.5) not to make part of the mask (otherwise the mask would not be circular but squared).