3D Game Engine & Tooling

I talk about tooling, 3D game engine development, and build servers.

July 3rd - Kanban.py

Since the 1st of this month I’ve been focusing on getting the release build working. I’ve finally got it in a really good state where I can click a single bat file, and after 3 minutes the release build executable is made.

Now that the build system is working as intended again. I’m focusing on moving everything from ClickUP to my own personal offline kanban solution.

I’m sure I’ll phase this out with a real kanban solution, maybe Obsidian Note Editor has kanban plugin. For now I’ve made my own tiny solution in python.

I have another 120~ marked completed tasks from the 3D Game Engine Editor project. Bringing this up to 500 trackable tasks completed since December of 2024.

I’d like to say that’s about 100 per month, but that would just be bad math. I’m going to say it anyways though cause it feels good. Maybe it’s realistically more likely about 60. Leaving us at 2 tasks completed, per day, on average. Cool.

I’ve been busy with the game. I’ve been slacking writing these notes to try to make up for lost dev time on the 3D game.

Like I mentioned earlier, I originally had about 500~ or so tasks from ClickUp that I’ve transferred over to local file management. I’ve used several project management suites previously:

• ClickUp
• Tello
• Monday.com
• Jira
• Asana
• Bugzilla
• Bitbucket (Issues & Wiki)
• Github (Issues & Community Tab)

And all of them I will continue to use them if I’m forced, since I’m the only team member on this project, I can use what I want, and I’ve decided to make it easier for me.

This project does not need anything complicated, or feature packed. I want a simple kanban view of all my lists, categorized by the folder they are in.

ClickUp was the closest to achieving that goal, but it had it’s own pitfalls that made me realize I was better off with my data being stored in git than having web based cloud systems.

I tried a few offline plugins that integrate with Obsidian, the software I use for managing my story writing. But none of the kanban plugins actually worked the way obsidian worked. Instead they required everything to be monofile, very odd. I suppose they want one dimensional lists with just names, but I want descriptions, dates, and more. I have a whole system already designed in my head, so I figure I’d just continue to use that.

This leads me to my next completely un-necessary invention - Kanban.py. Sure, I could have used something else, but I had a lot of fun making this quick app.

It’s a simple tool designed to provide everything that I used in ClickUp out of the box. It’s fully compatible with my flat-file approach of multiple files containing my data, and it integrates with my existing static blog builder. So I could publish everything public very easily if the community wants that.

ClickUp

Kanban.py

Of course Kanban.py lacks the cloud features, synchronized editing, mobile apps, etc. However it more than makes up for it with being able to sync with the git. I’d rather have less features and full git synchronization than have my project tracking be de-attached from my code. After-all my entire workflow revolves around markdown files for story outlines.

Right now I’m focusing on moving everything to be local files, so I’m re-organizing everything for the long term. Merging folders, TODOs and documents that I’ve accumulated since 2022 is a huge task but I’m almost through it at this rate.

I’m trying to now blend the 3D Game Editor aspect with the OS assets, this includes moving my auto build for the 2D cache to either ignore the root ‘assets’ folder, or I need to come up with a new folder name for the assets in the OS itself. Then I can move everything into the proper folders, everything is currently in a stable state but there is a lot of clean up I need to preform.

In other news, the release build version of the game ran for 12 hours and experienced no issues. Capping at 250mb memory, 20% gpu and 6% cpu while idling. These numbers don’t mean anything as my computer specs don’t match yours, but take my word for it that it’s pretty cool.

July 5th - SM64 / MEMORY.CRASH

The SM64 game engine has been renamed to MEMORY.CRASH game engine, kind of. The SM64 was just an internal working name. I’ll probably rename it again for the game release, but for now, I like it.

And it’s finally ready - I wrote a small credits cinematic I’ve had in my head for ages. It came out exactly how I pictured it. I could have done better and will continue to refine the tools, but I am extremely happy with the results.

I don’t think I will actually use the credits sequence I wrote, but I really enjoy the fact that I can imagine something cinematic, and I can now have the tools to actually direct a visual representation now.

I’ve been doing a lot of document house keeping, all of the 3rd party licenses are in order and the technology stack is finalized. I’m trying to move forward as if each stage is final, ready for release, within reason. The game demo related stuff will remain demo stuff so I can continue to progress without worrying about assets, those I will replace as I need to.

Since the repo has been growing I’ve decided to write a tiny repo tracking system called repo-health.sh:

It works by accounting for the the operating system usage, and calculates the active repo disk space against our total free space.

It’s slightly inaccurate for speed sake, I’m accounting for the operating system to increase with future updates, so 0.09gb is reserved. The real repo size is roughly 1.6gb.

This is perfect for my needs though, in one quick check I can see where we are at disk space wise. Previously I was relying on df -a which is great but I wanted something tailored and custom built - something that seems to be the theme for this month.

Some assets used are not created by me, and will not be in the final game, this is developer footage only

July 9th - Development Backwards

Following my development backwards rule (stolen from Valve - the last thing I should make, is the first level) - I’ve started with the credits. A simple idea I’ve had that I’m sure will change for the final game. This thankfully contained enough technical challenges that I’ve been able to refine the system more. I really enjoy the way I handle loading now. If something will load in and cause the main render to lag - for example, I just recently added a text markup processor into the Text Component, so for the credits it would cause lag when I introduced the Game Object.

My solution is now to just throw it in a background task. Since the Game Object list is a map, it can be cleanly removed or added to on demand, so all loading & processing can be done on another thread, then I simply pass the fully built Game Object into our rendering scene.

The result is a seamless loading mechanism. I assumed there would be some pitfalls to this, but so far it’s been fine. I was actually shocked at how well this solved the issue.

Originally, I thought I was going to have to iterate each process one by one, so for each 60 update frames per second, I could fill in 60 text generation frames. I wrote this and it was significantly less laggier, but it was still obviously introducing frame rate drop.

I’ve worked with background loading mechanisms before but I never though to add them into this project yet. Everything until this point has been loaded when the scene starts, now that I’m working on dynamic content, I need to have solutions for those problems.

Thankfully they have been extremely straight forward to write and allows the engine to follow my strict Windows PC Only Philosophy

I’ve thought about porting the 3D game engine games into another engine down the road. Since the entire scene is just a json blob it should be easy, I would just have to port over my specific runtime. The editor doesn’t need to be ported over, that wouldn’t make much sense anyways.

There is a pathway but I’m focused on getting this game out. Part of that includes the website so I’ve been working on the manual. After I ported everything over from ClickUp I started to get the base of the website and manuals started a bit more. It’s now in a good state where I just need to spend some time writing them when I get bored of the code.

July 12th - Borrowing From Unity

I’ve decided to try working on the final part of the editor that I’ve been wanting to borrow from Unity. Essentially Unity has a concept where you can map properties to be editable in the GUI:

public class Car : MonoBehaviour
{
     public string m_Make = "Toyota";
     public int m_YearBuilt = 1980;
     public Color m_Color = Color.black;
}

Code and image from unity3d.com.

I have always thought for years, this was one of the “killer features” unity had. This, and drag n drop asset placement. The two things that really make a modern game engine.

I decided I only need to focus on 4 major types:

• STRING
• BOOLEAN
• INT
• FLOAT

And containers for each type:

• ARRAY*
• LIST*
• MAP**
• • = Requires a variants of STRING, BOOLEAN, INT & FLOAT.
• ** = Requires variants of STRING + BOOLEAN, STRING + INT, STRING + STRING… ETC.

This ultimately ends up covering most of our variable types. LONG could be added in the future, but for now this will work for most cases I’ll encounter.

If you look closely at the icons, I’m also borrowing from Godot.

July 13th - Spare PC

I have an old PC from 2014 - I’ve been planning on restoring it for a while, but I haven’t had the need for it. Until lately, since I’ve been developing this game on my dev machine, compiling the release build at the same time I do anything else - really sucks

I can avoid compiling for testing, but if I want to see how something will happen for everyone who is actually going to play the game - I need to compile a release build of it. This means I can’t really use it for anything when I’m in the middle of a build - normally 2-3 minutes on average. Not an major issue by any means, but enough is enough.

I just finished swapping out the old components, with a new SSD it’s working like new again. I assume the drive went bad, cause after a quick swap it was working without issue. It’s not that great by todays standards, but it can run 64-bit windows and compile the game. So the extra compile time waited is fine as this allows me to verify the build worked fine by passing a -test parameter into the command-line during runtime. If it returns ’true’ then the automated build system knows all of the runtime tests passed and no errors were encountered. Then, we gain the obvious benefit of It removing the stress from my personal computer, so I can automate the build process.

This is something I’ve been trying to do while keeping it entirely offline. I don’t have the bandwidth to download the results from a remote service. So I wanted an all in one solution - keep my development and release builds local.

Eventually I could port the system over to some kind of GitHub-actions like concept, but for now I just want a low level system that alerts me via email if the stuff I pushed actually broke the release version unexpectedly.

July 14th - Build Server

I’ve been working on the remote build PC, so far it’s required a few changes to the way I manage the project to get it to build, but I got it passing the first of 2 stages. I had to fix a couple build errors that ended up being runtime bugs I wouldn’t have encountered until I hit them during runtime. So that was a headache I’m glad I avoided. This build server acts as a secondary testing grounds, so that’s a huge bonus. I almost completely overlooked this point as I was hyped focused, as I was just happy to move the remote builds from my local PC, as it’s extremely intensive.

I’ve basically remade GitHub actions, but entirely locally. It pulls the git, compiles, runs the tests, and reports the results via email. If there are any issues it will attach the error log.

July 17th - Auto Build Stack

The build server is working well, I’ve just been polishing it while I work on smaller bugs with the OS. Any time I push to the server I get an automatic email giving me analytics on the current build, and the previous build. All of these are the final release builds so it feels really good to have this piece of the puzzle automated and ready.

I made a small graph showing the way the stack currently works. I’ve got two servers isolated, one for email, web hosting, etc. The other is for hosting our Git server, with all of the assets, etc.

My workflow involves pushing to the git. Then my build server pulls from the git, makes a binary, then reports back the results of that attempt onto the mail server. I check the mail server regularly, and it’s setup on my phone. So I get alerts within 10 minutes. It takes a while to compile the game, so I just use this as a secondary check. Eventually I want the runtime test to do the real heavy lifting, but for now it’ll just be a planned future feature.

July 18th - Variable Mapping

I’ve been working on the new scripting system with dynamic properties. Kind of like how Unity has annotations for allowing variable default values to be edited in the GUI. I’ve tried to create something similar.

The end result is pretty sweet, I wrote a proof of concept a few days ago and now I’ve finally worked on implementing it into the actual engine itself. This has required me to change up how I do script initialization.

Previously script objects would be created every time a level loaded or restarted. This worked fine when I set the default values within code, but now that I’m loading them on level load, I have to handle it slightly differently. Instead of treating the scripts as killable objects, they need to live for the entire time the scene object exists, then I can reference the variables during loading, or saving as needed.

So this solved a few issues I had, and ended up making it so we could technically re-use the scene save as a persistent save for the actual game itself. Everything in the runtime scene should serialize and load in fine. Within reason. Obviously it’s not optimized for this kind of system, the idea was never store runtime data. It needs more work so I can signal specific data to be non-serializable. Other than that, it’s technically possible as is, which is cool.

The editor portion I still need to finish, I have a proof of concept I wrote that I’ll slide into the main editor GUI then I’m done. I’ve got an idea on how I can do scripts. Instead of it being drag and drop (since scripts aren’t file based) I can just have a text input prompt, then you write in the full script path. I’ll add autocomplete or something so it’s not too bad.

Then the properties will auto populate (the ones you can edit) and you’ll be free to make the changes you want. All of these will serialize and replay without issues.

Essentially, what I’ve created is a simplified version of what Unity provides. I’m skipping annotations as I don’t think it provides anything more than a pollable variable would. Maybe type-checking during compile time, but I’m not worried about those sort of benefits.