architecture - How to nest one Unity project into another?

I'm creating an AI course in Unity. With regard to my question, there are two important properties of the course:

1. Each tutorial is a separate Unity project that can be loaded up, allowing the student to start anywhere in the course and have a project that mirrors the project covered in the video they're watching.


2. Each tutorial builds off of previous tutorials. For example, tutorial one is just the base world provided with the course. Tutorial 5 includes all the code and assets introduced in tutorials 1-4, from the base world up.

My question is, how do I manage the nesting of Unity projects, without a lot of manual updating and copying?

Say, for example, when I'm creating the content for tutorial 5, I find a bug in the code created in tutorial 2. I don't want to have to update tutorials 2,3,4 and 5. I want to fix the bug in tutorial 2 and have it propagate through all the future tutorials that utilize that code or asset.

Essentially, each tutorial is a milestone on the way to the completed project. The completed project is the sum of all the tutorials and will contain all of the code, assets, prefabs, etc.

You can think of this like a dependency tree. Tutorial 5 depends on the code and assets of tutorial 4, which depends on the code and assets of tutorial 3 and so on. If this were a Visual Studio project, I'd create a new project and configure the dependencies under one solution.

_{In this example, the lines represent files that are identical to each other. Changes to the file in its origin project should update the file in subsequent projects. Ideally, a change to the file in any project would update all the others, but it's not a required feature}

I am, of course, using source control as well. So there may be a trick or two I can utilize there, but I'm not sure.

Some things I don't think will work:

1. Exporting code as a DLL, to be used in the next project. I don't think this will work because I don't want the student to have to do this with their own project. That means a student going through the course one tutorial at a time would have a different code structure than the student who started part way through.


2. Creating Unity packages. This would maintain the code, assets and prefabs, but (as far as I can tell) would require me to re-export and re-import packages for each project subsequent the changed project. But I believe this has the most potential if there's a way to automate the process.

Nesting projects like this would also be useful in other situations. Like a developer who's built up a core project they want to start all of their projects from. They could have 5 game that all have the same "base project" and each of the 5 game would benefit from bug fixes or feature adds that go into the base project, meaning the change is only made once.

Answer

The new Unity Package way

Unity has recently introduced a Package Manager. This is a system that most Unity components will be moving to, and it also allows you to create your own packages. Interestingly enough, these packages can be the entire contents of one Unity project, allowing you to nest one Unity project inside of another. There are two requirements:

1. 
   

   In the nested package, include a package manifest file in the directory you define as a package, this is in the form of a package.json file. Everything in the same directory as your package.json becomes part of the package. See "Package Manifests" under the advanced packaging section of this document

   
   


2. 
   

   In the encapsulating/parent Unity project, add a reference to your nested package in the manifest.json file. The manifest.json defines all the packages for your Unity project. At the moment, it must be manually edited to include this reference, but future UIs of Unity's Package Manager may include the ability to do this in Editor. See "Project manifests" under the advanced package section of this document.

   
   

When those conditions are met, you'll see the contents of the nested package included as a package in your parent package. This is superior to the below "The Unity way" because it's bidirectional. Change made to your nested package scripts/assets are made in the nested project, regardless if they're made from the nested Unity project Editor or the parent Unity project Editor.

I have see some issues with this strategy in regards to debugging. It's possible this is simply because the packaging system is still early in development. Sometimes when debugging scripts in the parent package, stepping into a class that's defined in the nested project will instead step into the script assembly Unity compiled for the package. This means only compiled meta data is available for the class instead of actual source (even though it's right there).

The Unity way

Automate the export and import of package files. This can be done using the command line options for Unity and a batch script file, something like the following script (line breaks added for readability):

Unity -projectPath "C:\Code\AI\Base\BaseWorld"
      -exportPackage "Assets\Scripts" "C:\Code\AI\BaseWorldPackage.unitypackage"
      -quit
Unity -projectPath "C:\Code\AI\Ch01\Chapter1"
      -importPackage "C:\Code\AI\BaseWorldPackage.unitypackage"
      -exportPackage "Assets\Scripts" "C:\Code\AI\Chapter1.unitypackage"

      -quit

This works great for selecting assets you want to be carried forward from previous projects. Each project adds its own assets, plus the assets imported from the previous project and produces its own package to be consumed.

The caveat here is that it's one-way. It's very easy to change a file in ProjectN+2, even though that file comes from ProjectN. That change will be overwritten the next time the script runs, because the change wasn't made in ProjectN, and all the updates only go in one direction.

The file system way

Hard links. Create hard links for each of the resource files you want to carry forward. This allows file content to be synchronized in two directions, because any edits to the file are essentially editing the same file. There are tools that can assist in the creation of these links to make the process easier.

The caveat here is that these structures are not easily managed by source control. Some source control systems don't work well with hard links or symbolic links. Additionally, they won't recreate the link structure when cloning a repository. Also, file renaming isn't well supported here.

The source control way

Utilize subrepositories or svn:externals. These are essentially one repository nested inside of another. This can get tricky, since these structures are commonly at the directory level. This means you can't add other files to the directories being nested. It makes for a somewhat of a splintered project. Additionally, to get the latest updates, source needs to be checked in and then out to the nested repos.

The robocopy way

Create a script to copy the files. Robocopy has a number of options for this. The script below has the following properties: 1. Copies all files and directories from BaseWorld to Chapter1 2. Synchronizes the contents of the contents of files between BaseWorld and Chapter1 (depending on which is newer) 3. Allows additional files to be in Chapter1 and don't get copied back to BaseWorld

robocopy "C:\Code\AI\Base\BaseWorld\Assets\Resources"
         "C:\Code\AI\Ch01\Chapter1\Assets\Resources"
         /E /XO /XF *.meta
robocopy "C:\Code\AI\Base\BaseWorld\Assets\Scripts" 
         "C:\Code\AI\Ch01\Chapter1\Assets\Scripts"
         /E /XO /XF *.meta

robocopy "C:\Code\AI\Ch01\Chapter1\Assets\Resources" 
         "C:\Code\AI\Base\BaseWorld\Assets\Resources"

         /XO /E /XX /XL
robocopy "C:\Code\AI\Ch01\Chapter1\Assets\Scripts" 
         "C:\Code\AI\Base\BaseWorld\Assets\Scripts"
         /XO /E /XX /XL

The caveat here is that it doesn't really support file renames. Renaming a file or directory will result in both the old copy of the file and the renamed file existing in subsequent directories.

Currently, there doesn't appear to be a single solution to make the process easy and transparent.

---

Final solution

I ended up using the robocopy way. It's two steps, the copy back and the copy forward.

Copy back. I start by copying any files that have been modified in project N to project N-1. Only files that exist in project N-1 are copied back from project N. This means files and directories that exist in later projects, don't get copied back to previous projects. And it means I can make changes to files in whatever project I happen to currently be using, and it will be updated all the way back to the project it originated in. Meta files are ignored in the copy back.

Copy forward. After copying back, I copy everything forward through the projects. This is a bit more liberal, if a file exists in project N-1 it will be copied forward to N. This ensures that any changes that were copied back from a project in the middle of the chain of projects, they'll now be copied forward to the latest project. Meta files are copied forward. Additionally, .asset files are copied forward too. This includes project settings, layers, sorting layers and tags.

This two step approach ensures that anytime a file changes, it's updated throughout the project chain. There were some growing pains when refining which file types to include and which to exclude. The project settings should make meta files visible and in plain text.

I found this approach to be the most seamless. I can run this script even when I have Unity projects open (unlike using Unity packages). And it's much faster than exporting and importing packages using Unity. I don't have to check in code or take a lot of time to create new linked files. I've been pretty happy with it so far. I have about 10 projects linked in this way right now (with dozens more to come). The script takes about 8 seconds to run through.

Implementation details

I created a little C# project to handle the execution of the robocopy calls. I tried doing it with batch scripts, but that was more work than it was worth :). I pass in the root directory of my projects. They're organized like so:

Root
  Ch01
    01 Project
    02 Project

    03 Project
  Ch02
    01 Project
    02 Project
  Ch03
    Etc.

I parse this structure to create a list of projects to iterate through.

Copy back, for each project, starting at the tip and moving to the base:

for(int i = projects.Count-1; i > 0; i--)

    robocopy projects[i] projects[i-1] /XD "Library" /XO /E /XX /XL
         *.cs *.shader /XD Library /XF *.asset *.unity

Copy forward, for each project, starting at the base and moving to the tip:

for(int i = 0; i < projects.Count - 1; i++)
    robocopy projects[i] projects[i + 1] /E *.cs *.prefab *.controller *.anim
         *.png *.mat *.shader *.meta  /XD Library
    robocopy projects[i]\ProjectSettings\ projects[i + 1]\ProjectSettings\ *.asset