architecture - How should I structure an extensible asset loading system?

For a hobby game engine in Java, I want to code a simple but flexible asset/resource manager. Assets are sounds, images, animation, models, textures, et cetera. After a few hours of browsing and some code experiments I'm still not sure how to design this thing.

Specifically, I'm looking for how I can design the manager in a way so that it abstracts away how specific asset types are loaded and where assets are being loaded from. I would like to be able to support both file system and RDBMS storage without the rest of the program needing to know about it. Similarly, I would like to add an animation description asset (FPS, frames to render, reference to the sprite image, et cetera) that's XML. I should be able to write a class for this with the functionality to find and read an XML file and create and return an AnimationAsset class with that information. I'm looking for a data-driven design.

I can find a lot of information on what an asset manager should do, but not on how to do it. The generics involved seem to result in some form of cascading of classes, or some form of helper classes. However I haven't seen a clear example that didn't look like a personal hack, or a point of consensus.

Answer

I would begin by not thinking about an asset manager. Thinking about your architecture in loosely-defined terms (like "manager") tends to let you mentally sweep many details under the rug, and consequently it becomes more difficult to settle on a solution.

Focus on your specific needs, which appears to be to do with creating a resource loading mechanism that abstracts the underlying origin storage and allows for extensibility of the supported type set. There's nothing really in your question concerning, for example, the caching of already-loaded resources -- which is fine, because in keeping with the single-responsibility principle you should probably build an asset cache as a separate entity and aggregate the two interfaces elsewhere, as appropriate.

To address your specific concern, you should design your loader so that it doesn't do the loading of any assets itself, but rather delegates that responsibility to interfaces tailored to loading specific types of asset. For example:

interface ITypeLoader {

  object Load (Stream assetStream);
}

You can create new classes that implement this interface, with each new class being tailored to loading a specific type of data from a stream. By using a stream, the type loader can be written against a common, storage-agnostic interface, and doesn't have to be hard coded to load from disk or a database; this would even allow you to load your assets from network streams (which can be very useful in implementing hot-reloading of assets when your game is running on a console and your editing tools on a network-connected PC).

Your main asset loader needs to be able to register and track these type-specific loaders:

class AssetLoader {
  public void RegisterType (string key, ITypeLoader loader) {
    loaders[key] = loader;
  }


  Dictionary loaders = new Dictionary();
}

The "key" used here can be whatever you like -- and it need not be a string, but those are easy to start with. The key will factor in to how you expect a user to identify a particular asset and will be used to look up the appropriate loader. Because you want to hide the fact that the implementation might be using a file system or a database, you can't have users referring to assets by a filesystem path or anything like that.

Users should refer to an asset with a bare minimum of information. In some cases, just a file name alone would be sufficient, but I've found that it's often desirable to use a type/name pair so everything is very explicit. Thus, a user might refer to a named instance of one of your animation XML files as "AnimationXml","PlayerWalkCycle".

Here, AnimationXml would be the key under which you registered AnimationXmlLoader, which implements IAssetLoader. Obviously, PlayerWalkCycle identifies the specific asset. Given a type name and a resource name, your asset loader can query its persistent storage for the raw bytes of that asset. Since we're going for maximum generality here, you can implement this by passing the loader a means of storage access when you create it, allowing you to replace the storage medium with anything that can provide a stream later on:

interface IAssetStreamProvider {
  Stream GetStream (string type, string name);
}


class AssetLoader {
  public AssetLoader (IAssetStreamProvider streamProvider) {
    provider = streamProvider;
  }

  object LoadAsset (string type, string name) {
    var loader = loaders[type];
    var stream = provider.GetStream(type, name);

    return loader.Load(stream);

  }

  public void RegisterType (string type, ITypeLoader loader) {
    loaders[type] = loader;
  }

  IAssetStreamProvider provider;
  Dictionary loaders = new Dictionary();
}

A very simple stream provider would simply look in a specified asset root directory for a subdirectory named type and load the raw bytes of the file named name into a stream and return it.

In short, what you have here is a system where:

• There is a class that knows how to read raw bytes from some kind of backend storage (a disk, a database, a network stream, whatever).


• There are classes that know how to turn a raw byte stream into a specific kind of resource and return it.


• Your actual "asset loader" just has a collection of the above aboves and knows how to pipe the output of the stream provider into the type-specific loader and thus produce a concrete asset. By exposing ways to configure the stream provider and the type-specific loaders, you have a system that can be extended by clients (or yourself) without having to modify the actual asset loader code.

Some caveats and final notes:

• 
  
  

  The above code is basically C#, but should translate to just about any language with minimal effort. To facilitate this I omitted a lot of things like error checking or properly using IDisposable and other idioms that may not apply directly in other languages. Those are left as homework for the reader.

  
  


• 
  

  Similarly, I return the concrete asset as object above, but you can use generics or templates or whatever to produce a more-specific object type if you like (you should, it's nice to work with).

  
  


• 
  

  As above, I don't deal with caching at all here. However, you can add caching easily and with the same kind of generality and configurability. Try it and see!

  
  


• 
  

  There are lots and lots and lots of ways to do this, and there is certainly no one way or consensus, which is why you haven't been able to find one. I've tried to provide enough code to get the specific points across without turning this answer into a painfully long wall-of-code. It's already exceedingly long as it is. If you have clarifying questions, feel free to comment or to find me in the chat.