Or "yes, it supports mods"

The October 2020 demo (corresponding with CommonCore 2.0.0 Preview 16) adds one big feature that I’ve wanted since the beginning and will probably never get used: addon support! If you’ve been following my Twitter, you’ve probably seen me tease it a bit in the past month or so.

The very first “eureka moment”, a literal “hello world”

I’ll start with a bit of background and maybe a minor rant. If you’re familiar with Unity you can probably skip this. Unlike some other engines, Unity is pretty firmly designed around the paradigm of starting with a whole project, building it into a black box of a package, distributing that and never touching it after the fact. This was fine at first, and is still absolutely fine for small projects, but it can be a problem for large projects, is definitely a problem for things like DLC and updates, and is completely unconducive to any sort of mod or addon functionality.

The smart folks at Unity have recognized this is a limitation, and there are a few mechanisms for extensibility. AssetBundles and the newer Addressable Assets allow a project to be broken up into pieces and parts loaded in at runtime. However these still assume you’ll have the entire project to begin with, which isn’t the case for addons. AssetBundles are still useful to us and I’ll bring them up later, but they’re not a solution in and of themselves.

The long and short of it is that we’ll have to work against the grain a bit and do some stuff that’s kinda hacky, and that addon support will never be as nice as it would be in an engine that supports it at a low level. However, it’s not impossible. There are many Unity-based games out there that are moddable to one degree or another.

At a bare minimum, I figured I would need to implement a few things:

• A way to load code at runtime
• A way to load resources at runtime
• Some way of executing loaded code at well-defined points

From there, in theory addons could do just about anything, though it would be entirely up to addon creators to figure out how. If I was really mean I could even omit the resource loading and leave that up to other people, but given how I’d set up resource management in CommonCore that wouldn’t make a lot of sense.

Past the bare minimum is:

• Provide convenient ways for addon creators to manipulate the game

That’s not really the focus here, at least not yet, but I’ll come back to this at the end.

Addon support has been a long-term goal and I’ve been working slowly towards it from the beginning. A lot of the “business logic” of the game loads its definitions from json assets and quite a bit of stuff prefers loading resources at runtime to setting them up beforehand. Both these approaches are less efficient and more awkward than setting up as much in the editor as possible, but make it a lot easier to change things after the fact. The Scripting system provides hooks to execute bits of code at well-defined points during the game, so that’s already one thing taken care of.

Finally getting addons working was my main technical goal for 2020. I knew there were some changes that needed to be made to the core code before I could actually get to the addon stuff. If you want to follow along you can view the code on GitHub. Be warned: a lot of it is gross.

Originally, CommonCore did all its startup before the game got going, using RuntimeInitializeOnLoadAttribute, and did it all in one go before the first frame. This was great because we could guarantee everything was loaded before the game logic tried to use it, but precluded a proper loading screen and precluded any kind of async. I’ve had my eye on changing this for a long time, but it’s never been critical. For addon support, though, we need async, not just to preserve players’ sanity but because some loading methods are async only.

It was surprisingly easy to implement, only requiring some minor changes and refactoring to our startup code in CCBase. I did leave the synchronous option in for platforms that don’t support it and for the editor. In a real game, we don’t have to worry about startup happening before game logic as long as the project is set up properly as we’ll wait at InitScene until CommonCore has finished starting up, but in the editor we usually start in the scene we’re testing. I added some hacky code that exits the scene, goes to InitScene, waits for startup then reloads the original scene. There are still some errors in the console from the first attempt at game logic loading in the scene, but it works!

There’s also an option now to load synchronously but do it after scene load instead of before, but it’s really the worst of both worlds and the only reason to use it would be for testing edge cases. I think that was the original reason I implemented it- just to test a few things- and then I left it in.

I thought I had to rework scene management entirely and build my own layer over Unity, but this didn’t end up being the case and I only had to make minor modifications to the stuff that enumerates the known scenes when I actually implemented addons. I also wanted to rework the order of event functions in various classes and add some CommonCore-specific stuff, but I didn’t need this and didn’t do it.

The big change was a complete rework of the way resources are managed in CommonCore. Resources are how assets are loaded at runtime. Originally they were synonymous with Unity Resources (loaded from the Resources folder with a path), then for a long time there was a thin layer that provided some minor redirection. Basically, that thin layer still called through to Unity’s Resources.Load but checked a few different paths to allow resources to be defined by the core and then overridden by the specific game.

I wrote a brand new ResourceManager. It’s a lot more complicated, and deserves its own article someday. It’s based around the idea of ResourceHandles representing assets that can be backed by Unity Resources, AssetBundles, files, or even created at runtime. The fixed overrides were replaced with a proper priority system. A lot of the complexity comes from the fact that we can’t just enumerate all the Unity Resources and have to wait until a path is tried. I really do wish there was an API for that. Other than that it is based around the idea of loading everything in at the start with some limited support for lazy loading.

The async load and new resource manager were added before addon support and left optional, and only changed to the default behaviour when addon support was added. At this point they’ve been tested enough that I’m somewhat confident they’ll work.

Most of this was done by the end of the summer. In the past month or so I turned my attention to putting together the last bits to actually load addons. This went surprisingly smoothly, and I think a lot of it comes down to all the pondering and preparation I’d done beforehand.

The structure of a CommonCore addon is inspired by various ways of structuring mods I’ve seen in other games over the years. 

An addon, at minimum, contains a manifest.json file defining the package name and some other metadata. I see this as being something that will do more as addon support is extended.

There are three folders in an addon with special meaning: elocal, expand, and managed. The resources in elocal are “mounted” at Addons/<package name>/ by the ResourceManager, while the resources in expand are loaded at the root and can override other resources. This makes it really easy, for example, to change a dialogue by just replacing its file. The managed folder contains assemblies which are loaded by the addon loader.

Note that modules must handle the resources loaded by addons themselves if, say, something they used was overridden. A new event method was added for this. This probably wasn’t the best design choice but I figured it was less likely to break things than moving loading later or making it all lazy loaded and rewriting half the loading routines.

I went with having addon packages referenced by package name rather than filename or path, and reverse-DNS notation as a convention for it. I don’t have any deeply opinionated reasoning for this choice, I just liked the feel of it. Addons are in folders rather than zipped packages which makes things easier for me but could cause issues with path lengths in theory.

There’s a lot of flexibility here. Resources can be loaded from loose files or AssetBundles. I didn’t realize assets in AssetBundles could have paths, so the loader is designed to load lots of AssetBundles each representing a flat folder. Honestly, I really wish AssetBundles were documented better. Addon load behaviour can be modified by defining a custom AddonBase derived class in the addon’s main assembly, which will be created and used instead of the default AddonBase.

The addon loader looks for addons in a few places, including the game’s folder and the persistent data folder. The load order is defined in the game’s config file, and the addon loader looks for each addon package and loads it in order.

Largely as a side-effect of the addon loading work, we can now load resources from StreamingAssets into the ResourceManager. I’m not sure how useful this will be but it’s there if I or someone else needs it. How practical is it to create an addon?

I can't resist the visual pun, sorry

Okay, so I did create a test addon as seen before. I knew that I’d run into issues and it wouldn’t be a smooth process. In fact, I think it was more annoying to put together than the actual addon support.

Some things are easy. Adding a dialogue was as easy as dropping a JSON file in the expand folder. Adding some graphics was a bit harder and required some work with AssetBundles, but it wasn’t too bad. Adding a scene, easy! Adding a scene that actually does stuff, hard!

I ran into one issue right away. I had no trouble importing UnityEngine or CommonCore assemblies into a .NET Standard library project created in Visual Studio, which made creating the first Hello World easy. You can also load the CommonCore assemblies into a blank Unity project as plugins, although you need all their dependencies as well and doing this feels incredibly janky. Most of the code is in Assembly-CSharp, though…

Which doesn’t load. I don’t actually get an error, but it doesn’t work. I suspect Unity doesn’t like having two Assembly-CSharps (one from the addon project) and just doesn’t know what to do here. So I had to figure out a way around that, which I’ll get to in a moment.

I also spent longer than I probably should have fighting with AssetBundles.

As I’ve mentioned before, AssetBundles are not well documented. I figured out by experimentation that paths are preserved in built AssetBundles, something which I wish I knew earlier on. I knew that export settings had to match for the AssetBundle, but I don’t think I learned this from the Unity docs, I think I learned it from StackOverflow. I knew that subassets were a thing and scene asset bundles were different somehow, but had to figure out the specifics of these things on my own and add special handling.

You don’t have to use AssetBundles. You can use loose files! Well, you can use loose files that an importer exists for. Some things were really easy to handle like text files, uncompressed audio, and textures, and these are built in. But I needed a way to extend this functionality- even after the fact- and I needed a way to load more complex assets.

To accomplish the former, I created an interface: IAssetImporter. This has a few methods, including CanLoad and Load (the actual names might be slightly different). The ResourceLoader maintains a list of IAssetImporters, and for each file it tries to load, checks the list of IAssetImporters and sees if any of them can load it.

To load more complex assets, we define them in a json file with the extension .jasset and a $assetType field inside. Note that that’s convention and I’m actually using IAssetImporter implementations to check and load these. The importer then reads the fields, loads other required files if necessary, and creates an asset. I’ve implemented this for Sprite (a Unity built-in type) and FacingSpriteAsset (one of my custom ones).

Another aside: I used a lot of mutable “context” objects being passed around in the addon load process. It was a convenient design pattern that made it easy to keep data handy, but feels a little scarier and more error prone compared to some of the more airtight techniques I’ve used elsewhere. I think it was a good fit here, and certainly better than some of the janky early CommonCore code.

Using a FacingSpriteAsset from a .jasset file

Okay, but what about the fact that we can’t access most of our game’s types from the code in an addon’s AssetBundle? You could write the code separately in a Visual Studio project, but you still have to hook it up somehow, and you’d run into the same issue. We have to proxy stuff somewhere and hook it up at runtime.

I created a new module, AddonSupport, which builds into its own assembly and does some proxying, just enough to allow you to create a SceneController in your scene, spawn some entities, and access some key utility methods from Assembly-CSharp.

By the way, you must use an Assembly Definition File in your addon so the resulting assembly isn’t another Assembly-CSharp and so that scripts are referenced by name instead of asset hash. You must also include that assembly in the addon, there is no code in an AssetBundle. Unity’s rules, not mine. 

The world's lamest Scene. But hey, loaded at runtime!

With all that done I was able to build a trivial addon that added some resources, ran a literal “Hello world!” and added a scene.

It’s clear that there are definitely some pain points, issues, and limitations with addon support as it stands. There’s a lot that could be done here, ranging from minor little things to massive overhauls. Extending AddonSupport or moving more stuff into Assembly Definition files would make scripting addons easier. One is extremely tedious, the other could break everything. It might be possible to do something with dummy assemblies but I’m kind of afraid to try it. Adding more importers would make using loose files more practical, which might be a better option than AssetBundles. Changing more things to use resources and definition files instead of being set up in the editor would make them more moddable, but also potentially more annoying to edit.

One thing I can dismiss pretty quickly is adding a separate scripting language. I’ve seen games do this, and it would make modding a lot smoother in many ways, but it’s just beyond my resources to add. While there might packages on the Asset Store that might make this a lot easier, I wouldn’t be able to include them and keeping CommonCore open-source and ready to go from a git pull is important to me, so I didn’t even look into it. I’ve already written most of my game logic without any of this in mind, anyway. If I’d intended to go this way from the very beginning I would have done all that differently.

I’m not sure how much of that I’ll ever implement, and how much I’ll leave as an exercise for the reader. For the time being, I’m just going to leave it mostly as-is. When I’m done with CommonCore 2.x and moving on to 3.x, I’ll take another look and re-evaluate. I think people need to actually try to use this to get an idea of what the best approach is, and we’ll see if that happens or not.

In addition to the pain points for addon creators mentioned above, there are some limitations and caveats to the addon support:

• It won’t work on IL2CPP and it won’t work without threading, which excludes basically every platform but standalone desktop.
• Everything is loaded at the beginning and kept in memory, so RAM usage will be pretty high with non-trivial addons.
• Loading is not optimized at all- it relies heavily on reflection and generates lots of garbage. Once everything is loaded the performance impact shouldn’t be too bad, though.
• The new ResourceManager is slower than the old nearly-native implementation. Technically not an issue with addon support but worth mentioning here.
• Addon code runs without restrictions. In my opinion you should always consider mods running with the same permissions as their containing process but I don’t even pretend here.

Because I know this will come up: why the term addons and not mods or plugins? I didn’t want to call them plugins, because that already has a specific meaning in the Unity world. I already have CommonCore tags and Unity Tags, and didn’t want to repeat anything like that. I was going to just call them mods originally, but mods sounds very similar to modules which is a term with a specific CommonCore meaning. Besides, not every addon will be a mod- I might use them internally. So addons it is.

My favourite “eureka moment”. Dialogue file and character sprite loaded at runtime.

I’ll end this with my usual disclaimer. This is my experience, this is how I did it, this is not a guide or tutorial. There are probably experts reading through this and cringing at every second sentence. And honestly, I’m not sure how addons will ever exist. But was really exciting to finally get a feature I’ve wanted for years actually working, and I learned a lot from the process.