OS GPU Rendering & Lua Integration

I talk about how the OS GPU rendering works and the issues I encountered, Lua Integration and the Chat App.

Feb 15th - OS GPU Renderer

I’ve started this development log, partially because I created a new editor to visually display design documents - but also to chart my progress.

The past few days have been bug fixing, as usual with game development. But these bugs are critical - the game is essentially still software rendering 2D entire UI - since the entire game is one big UI, this is important to note.

To utilize effects, we take this render, turn it into a gigantic screenshot - and display it using the GPU. This means the image can now be manipulated with shaders.

This solution has a lot of bugs it introduced that I’ve had to solve one by one. Most notability:

• Mouse, and keyboard interaction in general had to be programmed from the ground up.
• Fundamentally getting the concept working, there are two windows displayed at all times, and one of them is entirely invisible.
• I went through several iterations before I landed on the implementation we currently use. This R&D was fun but consumed a lot of time.
• Translating the mouse coordinates from the frame showing the GPU, to the frame rendering the UI.
• Mouse events, and keyboard events had to be translated from the visible frame to the invisible frame. This turned out to be a lot easier said than done.
• Menus in general refused to work and had to be managed manually. (Displaying, Hovering States, Recursive Menus)
• Text selection failed to show.
• Text Carets refused to work and had to be managed manually.
• Text carets failed to flash, when they finally were able to be shown.
• Popups wouldn’t show hovering states, and had to be managed manually.
• Cursors had to be monitored and swapped depending on the component hovered.
• Mouse scroll event needed to to forward to last hover component.
• Due to the new overhead from the GPU render, the intro timing had to be re-adjusted, so it only triggers after the GPU code is fully loaded.

Now that those bugs are fixed, we are exactly 5 weeks later from my original fulltime start (Started 01/06/2025). I created the original proof of concept back in 2023 - the desktop frame code came from a demo I had written in 2014 - but I shelved the proof of concept project.

The next steps for the game is to continue forward with the creation of the demo of the game. The text rendering is slightly blurry, bit I’ll fix this in a later update, for now the concept works.

Feb 18th - OS GPU Render Bug Fixing

GPU Bug: Mouse Translations

Mouse Translations from the GPU window to the CPU window is working perfectly now - dragging is fixed properly and I no longer have to include weird edge-cases into the code.

GPU Bug: MenuItems Fail To Click Action

This was a bit of a tricky problem, as the game is running two windows in the same process - the focus controller (or something) is causing the menu popups to close before the event can be processed:

• User Clicks
• Menu Hides On Window A
• Event Process Comes In from Window B, To Then process On Window A

As you can see, something is hiding the menu. The solution for now, is check if the last hovered component was a MenuItem. If it was, and we’re clicking the first click event (press), then treat it as a doClick action.

This has it’s own pitfalls (Such as click events should really only fire during release) but there is no way to capture release, we can only reliably do this once, immediately after the last MenuItem hover event. You could solve this by tracking more components, but it’s not really worth it, this works until I solve the focus bug.

GPU Bug: Blurriness

The GPU Mode, since it was created - had a blurry text bug - it would essentially cause every single text to be anti aliased.

This wasn’t a major problem, but clear text was basically impossible to show. I thought many times for several weeks how to fix it, until today it dawned on me. Compare width and height between the two frames (GPU and CPU).

Low and behold - they are different. I dug further and found the screenshot, and the canvas display window are different.

What if I trim the screenshot, so it’s exactly the same size as the canvas:

• The image now is no longer stretching.
• And the result is pixel perfect display, with pixel perfect mouse accuracy too.
• This fix introduces a small bug, but it was easy to fix. This was solved by adding padding around the monitor when running in the GPU mode. As a bonus this function can be re-used to introduce a padding functionality as a future game setting.

GPU Bug: Button Hover, Select & Click States

The theming system introduced some bugs, and one of them was an issue with button hover, select and click states wouldn’t display or change.

This was due to the Button UI code not correctly drawing the background as needed.

This has been fixed, but introduced some more issues, mainly I made the theme too simplistic, and now I’ve got colors clashing…Obviously I can fix this by extending the button theming, so that is probably what I’ll have to do.

I wrote a way to introduce theme updates without breaking previous themes by making the new variables change optionally - by using the old variables as a pointer if the variable isn’t set in the theme ini. Essentially maintaining backwards compatibility.

OS GPU Render Almost Done

The GPU render is preforming essentially 1:1 as the CPU render.

This looks convincing, and knowing that at any time, I can trigger a shader effect on demand and it looks and feels good. I think there is a lot I can do with this game.

It feels like a game demo now. There are a few small changes that need to be done (The button hovering code is kinda buggy, and theming in general needs more tweaking). But beyond that the core of the game is essentially where it was before I started the OS GPU rendering code.

Some assets used are not created by me, and will not be in the final game, this is developer footage only

Feb 25th - Chat App & Dialogue Scripting

Yesterday I wrote a small demo of the chat app and this lead into the next phase of development, the story beats. I figure it’s going to be narrative driven, so having the chat app and the script dialogue concept. I looked at existing implementations and although there are quite a few, I felt none of them would capture exactly what I’m looking for, as I myself don’t quite know.

I’m very fond of KCD1, and how the dialogue system works in that game. It reminds me of what fallout 4 was suppose to be in my eyes. I remember creating a dialogue system back in 2016, that had up to 4 dialogue options per dialogue. This to me was important, as I figured the more options meant the better gameplay it would have. Obviously that’s not true in hindsight.

So, what does this mean for our dialogue system? Not a lot. I’m going to experiment and we’ll see what that produces. I’m not going into this with any specific inspiration beyond the story I want to tell.

With that in mind, I’ve looked at the existing scripting languages, and I feel with my background, I could improve it so they are a bit more robust for my specific needs. This is a rough idea of what a syntax language could be - I based the original off of Ink:

=== intro ===

jerry: "\"Lorem Ipsum?\", I shouted at the computer."

1: "Choice A"
2: "Choice B?"
3: "Choice C"

$1:
jerry: "Lorem, lorem ipsum?"
computer: "Ipsum."
jerry: "Lorem."
"Ipsum lorem."
player: "What is going on here?"
> go_home

$2:
jerry: "...Really?"
computer: "Really."
> intro

$3:
player: "I'm leaving"
> go_home

=== go_home ===

This is lacking variables, comparative operations and tons more. But I thought this was a really good start. But ultimately this would mean creating a new interpreter, something I want to avoid. So I went looking for existing options.

Why Not Lua

After I weighted the pros and cons, and a few prototypes - I’ve decided to try going with Lua, I still need to finalize everything and see if this will work for the retail build - that is easier said than done. But I think this is the best option, and I’ve been meaning to include some kind of actual scripting capabilities into this. A sandboxed version of Lua is just the thing this operating system needs. I can lock down file IO, or remove it entirely.

Feb 26th - Lua Proof Of Concept

Today I’m working on the Lua integration, this means the Lua API should be ironed out first.

Lambda functions are something I’ve used for dialogue choices before, and I really like the way it works. Having a bunch of functions create actions in a queue, then slowly polling that queue is a great way of doing dialogue. As it let’s the person writing the script do it intuitively, without needing to think much. This allows us to insert dialogue choices as functions in a way that doesn’t disrupt the flow of story writing.

  jerry("Lorem ipsum "..gold..", is this enough for "..itemName(itemID).."?")
  computer("Consectetur adipiscing elit")
  jerry("Ut enim ad minim veniam")
  jerry("Quis nostrud exercitation ullamco")

  external_function(19383, "read value", externalVariableExample)
  
  giveItem(19383, "read value", playerID)

  option_1("Lorem ipsum dolor sit?", function()
    jerry("Duis aute irure dolor")
    jerry("Excepteur sint occaecat")

    option_1("Sed ut perspiciatis?", function()
      jerry("Nemo enim ipsam voluptatem")
      jerry("Quia voluptas sit aspernatur")
    end)

    option_2("Itaque earum rerum?")
  end)

  option_2("Lorem ipsum?")
  option_3("Sunt in culpa qui officia deserunt.")

I’ve created a small demo GUI that works to help test out the API before I merge it into the ChatApp.

Feb 27th - Syntax Refinement

I might make the prototype app I wrote yesterday work inside of the OS as a demo to test everything, then I can work on incorporating the API into the real chat app itself.

Here’s the syntax I ended up with:

  jerry "Lorem ipsum dolor sit amet, consectetur adipiscing elit?"
  computer "Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua"

  jerry {
    "Ut enim ad minim veniam"
    "Quis nostrud exercitation ullamco laboris"
  }

  external_function(19383, "read value", externalVariableExample)
  giveItem(19383, "read value", playerID)

  option_1("Where does this go?", function()
    jerry "Duis aute irure dolor in reprehenderit"
    jerry "Excepteur sint occaecat cupidatat non proident"

    option_1("What?", function()
      jerry "Sunt in culpa qui officia deserunt mollit"
      jerry "Anim id est laborum"
    end)

    option_2("I've had enough")
  end)

talkable = {}

function addTalkable(key, nickname, fullName, id, image)
  talkable[key] = {
    nickname = nickname,
    fullName = fullName,
    id = id,
    image = image
  }
end

function init()()
  addTalkable("jerry", "Jerry", "Jerry McBurger", "JBurg33", "img/icon/jerry")
  addTalkable("computer", "Computer", "Thomas McBurger", "CoMpuT3R", "img/icon/computer")
  updateTalkables()
end

function jerry(msg)
  message("jerry", msg)
end

function computer(msg)
  message("computer", msg)
end

Feb 28th - Dialogue Choices

Internal Dialogue Popups were a pain, something I didn’t think would be an issue. Lua functions call on my core functions, and I can just poll the talkables table I created in Lua as the map of all the current non playable characters who are going to converse with the player. It also includes player responses.

Here’s how the current syntax looks so far:

function entry()

  jerry "\"Lorem ipsum dolor sit amet?\", I shouted at the computer."
  computer "Consectetur adipiscing elit, sed do eiusmod tempor..."
  
  option({option_1 = {"Lorem ipsum dolor sit amet?", function()
      jerry "Ut enim ad minim veniam"
      jerry "Quis nostrud exercitation ullamco"
      
      choice_entry_1()
    end},

    option_2 = {"Duis aute irure dolor?", function()
    end},

    option_3 = {"Sunt in culpa qui officia.", function()
    end}
  })
end

The next steps for this are going to be choices, let’s see how far I get with that…

Choices are added, abusing the way tables work in Lua made this painless…Then I had a thought, KCD1 has greetings in their dialogue. “Hello Henry” - then the dialogue ’loop’ happens. (You can select dialogue options, or someone is conversating with you)

Lua allows method redefinition by just redefining the function. It’s that easy.

function entry()

  jerry "\"Lorem ipsum demo?\", Lorem ipsum."

  entry = function()

    choice({
      option_1 = {"Lorem ipsum dolor sit?", function()
        jerry "Ut enim ad minim veniam"
        jerry "Quis nostrud exercitation ullamco"
      end},

      option_2 = {"Dolor sit amet?", function()
        computer "Consectetur adipiscing elit..."
      end},

      option_3 = {"Sunt in culpa officia.", function()
        computer "Excepteur sint occaecat cupidatat!"
      end}
    })

  end
  
  entry()
end

Feb 28th - Month End Summary

290 Completed Tasks

As of writing, I’ve completed 290 tasks - beating my goal outlined in January of 100. This includes some massive milestones that I personally never could have imagined would have been a part of this project.

Themes Update

Account themes are finally fully defined, and this includes file overwrites, icons, configuration, color palettes and more. The system is fully flexible and designed in a way where themes can offer essentially complete control over the look of the game.

CRT Shading Update

CRT Shading is still a massive work in progress, but we have the ability to pass our CPU based GUI render through the GPU, allowing us to use shaders and other GPU-related code.