directx9 - What is a better abstraction layer for D3D9 and OpenGL vertex data management?

My rendering code has always been OpenGL. I now need to support a platform that does not have OpenGL, so I have to add an abstraction layer that wraps OpenGL and Direct3D 9. I will support Direct3D 11 later.

TL;DR: the differences between OpenGL and Direct3D cause redundancy for the programmer, and the data layout feels flaky.

For now, my API works a bit like this. This is how a shader is created:

Shader *shader = Shader::Create(
    " ... GLSL vertex shader ... ", " ... GLSL pixel shader ... ",
    " ... HLSL vertex shader ... ", " ... HLSL pixel shader ... ");
ShaderAttrib a1 = shader->GetAttribLocation("Point", VertexUsage::Position, 0);
ShaderAttrib a2 = shader->GetAttribLocation("TexCoord", VertexUsage::TexCoord, 0);

ShaderAttrib a3 = shader->GetAttribLocation("Data", VertexUsage::TexCoord, 1);
ShaderUniform u1 = shader->GetUniformLocation("WorldMatrix");
ShaderUniform u2 = shader->GetUniformLocation("Zoom");

There is already a problem here: once a Direct3D shader is compiled, there is no way to query an input attribute by its name; apparently only the semantics stay meaningful. This is why GetAttribLocation has these extra arguments, which get hidden in ShaderAttrib.

Now this is how I create a vertex declaration and two vertex buffers:

VertexDeclaration *decl = VertexDeclaration::Create(
        VertexStream(VertexUsage::Position, 0,
                                VertexUsage::TexCoord, 0),
        VertexStream(VertexUsage::TexCoord, 1));


VertexBuffer *vb1 = new VertexBuffer(NUM * (sizeof(vec3) + sizeof(vec2));
VertexBuffer *vb2 = new VertexBuffer(NUM * sizeof(vec4));

Another problem: the information VertexUsage::Position, 0 is totally useless to the OpenGL/GLSL backend because it does not care about semantics.

Once the vertex buffers have been filled with or pointed at data, this is the rendering code:

shader->Bind();
shader->SetUniform(u1, GetWorldMatrix());
shader->SetUniform(u2, blah);
decl->Bind();

decl->SetStream(vb1, a1, a2);
decl->SetStream(vb2, a3);
decl->DrawPrimitives(VertexPrimitive::Triangle, NUM / 3);
decl->Unbind();
shader->Unbind();

You see that decl is a bit more than just a D3D-like vertex declaration, it kinda takes care of rendering as well. Does this make sense at all? What would be a cleaner design? Or a good source of inspiration?

Answer

You're basically running into the kind of situation that makes NVIDIA Cg such an attractive piece of software (aside from the fact that it doesn't support GL|ES, which you said you're using).

Also note that you really shouldn't use glGetAttribLocation. That function is bad juju from the initial days of GLSL before the folks in charge of GL really started to grok how a good shading language should work. It's not deprecated as it has the occasional use, but in general, prefer glBindAttibLocation or the explicit attribute location extension (core in GL 3.3+).

Dealing with the differences in shader languages is by far the hardest part of porting software between GL and D3D. The API issues you're running into regarding vertex layout definition can also just be looked at as a shader language problem, as GLSL versions before 3.30 don't support explicit attribute location (similar in spirit to attribute semantics in HLSL) and GLSL versions before 4.10 iirc don't support explicit uniform bindings.

The "best" approach is to have a high-level shading language library and data format that encapsulates your shader packages. Do NOT simply feed in a bunch of raw GLSL/HLSL to a thin Shader class and expect to be able to come up with any kind of sane API.

Instead, put your shaders out into a file. Wrap them up in a bit of meta-data. You could use XML, and write shader packages like:

  
          glsl vertex shader code goes here
    ]]>
          glsl fragment shader code goes here

    ]]>
  
  
          hlsl effects code goes here
      you could also split up the source elements for hlsl
    ]]>
  

Writing a minimal parser for that is trivial (just use TinyXML for instance). Let your shader library load up that package, select the proper profile for your current target renderer, and compile the shaders.

Also note that if you prefer, you can keep the source external to the shader definition, but still have the file. Just put file names instead of source into the source elements. This may be beneficial if you plan on precompiling shaders, for instance.

The hard part now of course is dealing with GLSL and its deficiencies. The problem is that you need to binding attribute locations to something akin to HLSL semantics. This can be done by defining those semantics in your API and then using glBindAttribLocation before linking the GLSL profile. Your shader package framework can handle this explicitly, with absolutely no need for your graphics API to expose the details.

You can do that by extending the above XML format with some new elements in the GLSL profile to explicitly specify attribute locations, e.g.

  
    
    
          #version 150

      in vec4 inPosition;
      in vec4 inColor;

      out vec4 vColor;

      void main() {
        vColor = inColor;
        gl_Position = position;
      }
    ]]>

  

Your shader package code would read in all the attrib elements in the XML, grab the name and the semantic from them, look up the pre-defined attribute index for each semantic, and then automatically call glBindAttribLocation for you when linking the shader.

The end result is that your API can now feel much nicer than your old GL code likely ever looked, and even a bit cleaner than D3D11 would allow:

// simple example, easily improved
VertexLayout layout = api->createLayout();
layout.bind(gfx::POSITION, buffer0, gfx::FLOATx4, sizeof(Vertex), offsetof(Vertex, position));
layout.bind(gfx::COLOR0, buffer0, gfx::UBYTEx4, sizeof(Vertex), offsetof(Vertex, color));

Also note that you don't strictly need the shader package format. If you want to keep things simple, you're free to just have a loadShader(const char* name) kind of function that automatically grabs the name.vs and name.fs GLSL files in GL mode and compiles and links them. However, you're absolutely going to want that attribute metadata. In the simple case, you can augment your GLSL code with special easy-to-parse comments, like:

#version 150

/// ATTRIB(inPosition,POSITION)
in vec4 inPosition;
/// ATTRIB(inColor,COLOR0)
in vec4 inColor;

out vec4 vColor


void main() {
  vColor = inColor;
  gl_Position = inPosition;
}

You can get as fancy as you're comfortable with in the comment parsing. More than a few professional engines will go so far as to make minor language extensions that they parse out and modify, even, such as just outright adding HLSL style semantic declarations. If your knowledge of parsing is robust, you should be able to reliably find those extended declarations, extract the extra information, and then replace the text with the GLSL compatible code.

No matter how you do it, the short version is to augment your GLSL with the missing attribute semantic information and have your shader loader abstraction deal with calling glBindAttribLocation to fix things up and make them more like the easy and efficient modern GLSL versions and HLSL.