opengl - in the shadow of a sphere

(Related, but somewhat different, to my previous question)

How can I determine in a fragment shader if a fragment is in the shadow of a sphere?

That is, if it is occluded by the sphere and is past the sphere's horizon from the camera (if you are in front of the horizon you are not in the shadow even if you are in the sphere; the sphere is not solid)

In perspective, the horizon of the sphere is in front of the centre-point. Imagine holding a football at arms-length and stare at a point on the horizon of it; now move the football closer to you eye; what happens? It is no longer visible; the closer the sphere is to the eye, the less of the surface you can see:

As I imagine it, it is:

1. 
   

   are you in the cone that is from the camera and passes through the horizon of the sphere as seen from the eye? and

   
   


2. 
   

   are you past that horizon?

   
   
   

How do you compute the plane of the horizon, the cone, and how do you test for it in the fragment shader?

1. 
   

   Imagine you had a ray that was through camera and fragment. The nearest distance between that ray and the centre of the sphere being less than the sphere's radius would tell you if it was in the 'cone' of the sphere.

   
   


2. 
   

   Now imagine you knew the distance the camera to the horizon; if the closest point on the ray was less than this distance, its in front of the sphere; else its past the horizon. (We can make this simplification the fragments we want to test are never deep in the middle of the sphere.)

   
   

With these two values, you determine if a fragment is 'in the shadow' of the sphere.

But how do you compute this? What, even, is the coordinate of the camera (0,0,-1 if orthogonal projection, else 0,0,0?)? And how far away is the horizon of the sphere?

And what's the code for nearest point on ray to point? What I've come up with is [src]:

t = (P-B).(A-B) / (A-B).(A-B)

If P is the sphere's centre, and A is the fragment's position and B is the camera (at 0,0,0 so can be omitted as its a no-op):

// its a unit sphere:
var sphereCentre = mat4_vec3_multiply(
    mat4_inverse(mat4_multiply(pMatrix,mvMatrix)),

    [0,0,0]);
gl.uniform3fv(program.sphereCentre,sphereCentre);
gl.uniform1f(program.sphereRadius,1);

Then the vertex shader just has to pass the fragment position along:

precision mediump float;
attribute vec3 vertex;
uniform mat4 pMatrix, mvMatrix;
varying vec3 p;
void main() {

    gl_Position = pMatrix * mvMatrix * vec4(vertex,1.0);
    p = gl_Position.xyz/gl_Position.w;
}

And the fragment shader sees if its inside-the-cone using distance to sphere centre:

precision mediump float;
uniform vec4 fgColour, bgColour;
uniform vec3 sphereCentre;
uniform float sphereRadius; // always 1 in my game fwiw
varying vec3 p;

void main() {
    float t = dot(sphereCentre,p) / dot(p,p); // where on line?
    vec3 d = (p*t) - sphereCentre; // distance from nearest point to sphere
    //### now we need to know if its in front of the horizon to force fgColour ???
    gl_FragColor = (dot(d,d) <= sphereRadius*sphereRadius)? fgColour: bgColour;
}

This might be along the right track, but its not working (it looks hopeful drawn in ortho; in perspective it often draws things in the wrong colour). And how to compute the horizon?

Answer

To answer my own question:

The camera is at 0,0,0 in view space.

The sphere's centre has to be converted to view space and passed in as a uniform; this means multiplying it by the modelview matrix.

The vertex shader passes on the fragment's coordinate in view-space:

precision mediump float;
attribute vec3 vertex;
uniform mat4 pMatrix, mvMatrix;
varying vec4 pos;
void main() {
    pos = (mvMatrix * vec4(vertex,1.0));
    gl_Position = pMatrix * pos;

}

The fragment shader does the check:

precision mediump float;
uniform vec4 fgColour, bgColour;
uniform vec3 sphereCentre;
uniform float sphereRadius2; // always 1 in my game fwiw
varying vec4 pos;
void main() {
    vec3 p = pos.xyz/pos.w;

    float t = dot(sphereCentre,p)/dot(p,p);
    vec3 d = (p*t)-sphereCentre;
    gl_FragColor = (dot(d,d) > sphereRadius2 || t<=1.0)? fgColour: bgColour;
}

This computes where on the fragment's ray the nearest point is, as a ratio t.

It also computes the distance (squared - dot(d,d)) between the nearest point and the sphere's centre.

If the nearest point is beyond the sphere or the intersection of the line is less than 1 (meaning it is in front of the sphere) then it is in the foreground.

To compute point sprites correctly you need an extra check if it is in the background because the check outlined above is for the centre-point of the sprite and not each fragment in the sprite. If its possibly in the shadow of the sphere, you need to additionally check the exact fragment in the sprite.