Let’s start with the game. Lately, I’ve been exploring just how far I can push old-school ASCII RPG style game frameworks. My latest one is called “Thunder Lizard,” which procedurally generates a prehistoric island populated with dinosaurs fighting for dominance as an active volcano threatens the whole island. You can go play it if you’d like.

To render it in AI, the basic plan was to grab a frame from the game, run it through an image generation model, and replace the displayed frame with the resulting image, for every frame. This presented a number of challenges and requirements that lead me on a deep dive into the wide offering of cutting edge image generation models offered today. But first, let me show how it turned out.

The need for speed

The main constraint for real-time AI rendering is latency. Most games run at least at 30 frames per second (FPS) which only gives you 30 milliseconds to do the following:

• Connect (and authenticate) with an inference provider

• Transmit the prompt (including source image data)

• Wait generation to complete

• Receive the new image data and display it

Considering a normal latency to load an image can be a couple hundred milliseconds, this constraint seems impossible. However, fal.ai specializes in offering “lightning-fast inference capabilities” for generative media, including a few Latent Consistency Models (LCM) that approach 100ms generation times. To further minimize latency, fal.ai also offers a WebSocket connection to remove the connect and authenticate steps from subsequent requests. Finally, they offer the option to stream images as Base64 encoded data for immediate direct access.

By taking advantage of all of these optimizations, and by using 512×512px images, I was able to run at 10 FPS with around one second of latency. This is in fact the same frame rate targeted in the original game because it felt best with the ASCII visuals. I could likely go faster, but doing so would risk rendering images out of order if newer requests finished before older ones came in. The latency was the larger issue because it is quite noticeable as a “lag” between the source game and player inputs and the AI rendered frames, which you can see in the video above.

Choosing a model

Fal.ai has over 300 image generation related models currently! I found this quite overwhelming when I started. I spent quite bit of time learning about the different model providers and their specialties and capabilities. I generated hundreds of images in their online playground to understand how they each worked and which ones would work for my needs.

I narrowed down the list based on these hard requirements:

• Very fast generation times

• Source image adherence

• Decent look and feel

Only a handful of models were fast enough, so that part was easy. The challenge was getting a good enough output. Fast models by definition are distilled or compressed in some way, making them less powerful than larger, slower models. Not only does this effect image quality, but it also limits generation options, like using LoRA or ControlNet.

One of the big challenges was finding the appropriate way to make sure the output image “lined up” with the actual game frame to preserve the solidity of the game world. I explored a number of approaches, including image-to-image (i2i) models, ControlNet, and image editing models.

Layout consistency via ControlNet

I originally thought that ControlNet would be the way to go. ControlNet is a way to provide extra “guidance” to a model to control the layout of the generated image. It comes in many flavors, such as poses, outlines and depth. I was particularly focused on “segmentation,” which is a way to visually indicate “zones” for content to align with. Since I controlled the rendering engine of my game, I was able to output a version of the frame that seemed like a good segmentation ControlNet reference.

I had hoped to be able to use a text-to-image model with a prompt describing a generic top-down prehistoric terrain, and let the ControlNet guide the zones, like blue for water and red for lava. Not all models offered ControlNet, but a few did.

I was surprised to find out that this approach didn’t work very well at all. I think this was partially because I didn’t figure out how to correctly label the segmentation zones, and partially because the blocky terrain layout doesn’t carry enough meaningful visual semantics for the model. I experimented with other ControlNet types, like outlines, and had slightly better results, but ultimately gave up on this approach.

Layout consistency via image-to-image models

The other option was to use the game image as a source for a model that could do image-to-image transformations. Some models like FLUX Kontext are specialized for image editing, while others use the provided image as a prompt. I tried both approaches, with prompts like “turn this blocky image into realistic aerial terrain” or “aerial terrain of a prehistoric island.”

The resulting images followed the layout of game frame much better with this approach. However, it became harder to control the look and feel of the generated image. Too much image “strength” and the output would be blocky just like the input. Too little and it would look better, but not line up. The included text prompt had surprisingly little effect. Some models offered additional options such as IP adapters, style transfers, and LoRAs, which I experimented with as well.

The right look

I tried endless combinations of settings and techniques to get a look I liked. Here are just a few examples.

Ultimately, I settled with the best result I could get on the models there were fast enough for my needs. I used the fast-lcm-diffusion/image-to-image endpoint with the StableDiffusion 1.5 model and the prompt “top down, rpg terrain map for a game set in prehistoric times. red is lava, blue is water, green is grass / trees”. I learned that the blocky version of the game frame that I intended to use as a segmentation map resulted in a better i2i result than the ASCII equivalent.

I wasn’t very satisfied with the look, but it was the best I could do. I also found it disappointing that the models were never able to render the dinosaurs as sprites within the terrain, despite my best prompting efforts.

Before moving on, I did spend some extra time exploring ways to get a better look with a better model, even though I knew it wouldn’t work for real-time, just to see how far I could get.

LoRA

To get a better look, I tried a few “style transfer” models, but ultimately made my own “style” by training custom LoRA weights. LoRA is a technique to fine-tune a model in a lightweight and portable way. Here was my process:

1. I used the general FLUX model with the prompt “Aerial view of terrain. Realistic that has water, lava, trees, and grass” to find a style I liked.

2. I used the same model with variations of the prompt, “top down rpg game map of prehistoric landscape with water and dense trees and a medium dinosaur and some lava on the edge” plus the original image as a “reference image” parameter to take advantage of FLUX’s built-in generic style reference, to generate about ten sample images.

3. 

   I used the FLUX LoRA Fast Training model with my sample images to train directly on fal.ai, which only took a couple of minutes and cost less than a dollar.

4. I used the resulting LoRA weights on FLUX LoRA Image to Image with the prompt “tl_rpg terrain - blue is water, green is grass and trees, red is lava, crosses are dinosaurs” where “tl_rpg” is the trigger word from my training, plus the image frame from my game to get style consistent, source adhering, high-quality generated images.

Here are the results. Interestingly, the ASCII frames worked better than the segmentation versions, which was opposite from what I found with the smaller models. These still are not as nice as the generated images that aren’t bound to the source game frames, but I was unable to get any closer than this.

For what it is worth, I did try using this version for game rendering, though at a four-second latency, it was un-playable, which is a shame because it looks much better.

Putting it all together

With everything in place, it was time to integrate image generation with the game. I used the JavaScript SDK that fal.ai provides to connect via WebSockets. On every render call in my game loop, I actually drew to two separate canvases, one for the original output and a second hidden one that rendered the block version at 512×512 pixels. Then I captured the image data from the second canvas as a base64 data url and sent it with the rest of the prompt and settings as a message over the WebSocket connection. Upon the async response, I rendered the resulting image data into a third canvas for that purpose. You can see the actual code if interested, and the image below shows the three canvases side by side to demonstrate the three different renderings.

I would call this experiment a success. It only costs a few cents per minute with the fast model at the low frame size and frame rate. I accidentally ran the LoRA model above at 10 FPS and blew through $10 in a few seconds, so model selection matters!

One remaining issue is frame to frame consistency. While the models actually are quite stable, especially with a set seed and high image strength setting, you can still see “jumpy” differences where trees turn into rocks or grass with each new frame which is disorienting. I thought of a few techniques that could help with this. One is to use LoRA, though even if the image style is consistent, the exact placement of terrain features still jumps from frame to frame. A more advanced technique that could work in this use case is to use “out painting” to provide the previous generated frame shifted by the movement delta, the current source frame, and a mask over the new “empty” pixels. This might make a really impressive and smooth visual experience, though it currently wouldn’t be possible at the required latency with the models available today.

All in all, it is amazing to see AI image generation happen in real time. I am excited to see where this will go in the future. I also really like the idea of using AI to enhance or alter an existing source that controls the consistency of the underlying world. This project was a fun exercise to push the envelope, but I could imagine this being a powerful technique to quickly and easily experiment with multiple different styles from a single low-res prototype. Thanks for following along!

I use Leonardo for the image generation which offers some nice power tools. After playing around with different models and prompts to discover a general look that I like, I focus on the part of the prompt that is specific to the card I am working on. One particular card featured an "Angry skunk." This is the prompt that I used:

Design for a Tarot card: Angry Skunk, forest theme, muted colors

The first and last parts are for the overall theme and look, and the middle part is for the subject. This is what it made:

The prompt is very short and simple. I usually start there, then add to it to flesh out the details or composition. In this case, the skunk is way too cutesy and cartoony and also too large in the frame.

I tried to adjust it by changing the prompt to say "behind a bush," "in the distance" or "long shot." I also added "cartoon" as a negative prompt. But for whatever reason, the model really thinks that skunks ought to look cutesy, cartoony, and in your face. So I had to get more advanced.

The model was great at making general forest scenes, so I made a fallen tree trunk, then opened that image in Leonardo's Canvas Editor and started drawing where I thought the skunk belonged. Now I'm not great at illustration, especially on my laptop's trackpad, and skunks are very confusing to draw from memory. But I figured a few black and white blobs would give the model the general idea. The purple bars show the masked area for the inpainting model to focus on:

You can see from the header image of this post how the model progressed through a couple of iterations of making my blobs actually look like a skunk. By changing the inpainting parameters and approach I was able to go from "sketch2image" and then "image2image" to get the final version. It has the right look and feel and most importantly, it is positioned properly in the frame. Here is the final card:

Raccoons were another small rodent that the model had trouble with. Like the skunk, it wanted very cute in-your-face raccoons despite my best efforts. I tried to get creative by changing the concept to "raccoons with glowing eyes hiding in the trees," but it just didn't work out right. So I did a similar process and was very happy with the results:

This technique works great any time you want to control or modify a specific element in an image. It elevates the AI image generation experience from a prompt to a paintbrush and makes me feel more involved in the process and more enabled to express what is in my mind. I've also used it for practical tasks like cutting out a frame or constraining an ink drawing to a specific area.

This type of interaction will only get better. I saw a demo just the other day of a new AI technology called real-time image generation with low latency models. You can paint a canvas on the left, and an AI-generated rendition of your sketch renders in real time on the right. Try it yourself on fal.ai.

AI art is not perfect, but for prototypes like this where volume and expediency are key, this technique can be very helpful.

I gave a talk at the "AI Engineer Summit 2023" on how I accomplished this. You can watch it here or read the generated "transcript" below.

Also, the code is available on Github if you want to fork it to make your own game:

• Server/AI pipeline: https://github.com/jschomay/lost-game-ai-asset-server

• Client/game: https://github.com/jschomay/Lost

The Concept: Getting Lost in the Forest

The premise of the game is simple yet intriguing. As a player, you find yourself lost in a vast forest. Each scene presents encounters that impact your vigor and courage. The goal is to find your way back home before your courage runs out. A maze without walls. The only drawback was that I only made 16 scenes in a 4x4 grid, so after playing for a while they would repeat.

My favorite part of creating this game was generating each scene using AI and seeing what it came up with. I wanted to share this experience with the players and give them a unique scene every time they moved to a new location. By leveraging the power of AI, I was able to create a game of infinite exploration!

AI-Generated Scenes: A New and Unique Experience Every Time

To create the scenes, I used a JSON object that describes the initial state of the scene and its subsequent variations. Initially, I employed OpenAI's Completion Endpoint and prompt engineering techniques to generate new scenes via their GPT3.5 Turbo LLM. The prompt I used was highly detailed and specific, ensuring that the AI-generated scenes maintained the required JSON format and contained fitting, varied, and interesting content.

Initial prompt:

I'm making scenes for a game.  The setting is being lost in the forest.  Each scene has a unique feature that can be frightening, inspiring, relaxing, etc.  The scene can feature an encounter that is either animal, natural, human, or supernatural.
The player has two stats: vigor and courage.  Vigor is impacted by events that give or take energy and effort.  Courage is impacted by events that add or reduce fear and uncertainty.
A scene has title and 1-2 sentence initial description of what you see and how it effects you (in second person).  
It also has stat change of one, both, or neither of the player stats.  Changes should be an integer between -4 and 4.
A scene also has returning description that highlights any potential changes and how they impact you and a corresponding returning stat change.

Output format is a single JSON map:

{
  "title": "Intimidating Bear",
  "on_discover": {
    "description": "A massive bear stands tall, its powerful presence filling you with both awe and fear. You tread cautiously, knowing any sudden movement could provoke its fierce wrath. Best not to linger.",
    "stats": [
      {
        "stat": "courage",
        "diff": -2
      }
    ]
  },
  "on_return": {
    "description": "The bear is still here, or has it been following you?",
    "stats": [
      {
        "stat": "courage",
        "diff": -4
      }
    ]
  }
}

After obtaining satisfactory results, I fine-tuned the model using OpenAI's Fine Tuning Endpoint. I gave it 50 hand-selected generated examples. I also gave it a simplified prompt without explicitly mentioning the JSON structure.

Simplified prompt:

Help the user make up scenes for a game. The setting is being lost in a forest. Scenes have different feelings and involve various encounters.
The output must follow a specific JSON map.  The only valid stats are `vigor` and `courage`.

The simplified prompt would reduce cost and latency. I was impressed to see that I still got consistent and valid results. The fine-tuned model produced scene descriptions that aligned perfectly with my requirements.

AI-Generated Images: Creating Stylistically Consistent Art

In addition to generating textual descriptions for each scene, I also wanted to create stylistically consistent images to enhance the game's visual experience. To accomplish this, I utilized a tool called Leonardo.Ai, which not only allows image generation but also lets you create custom image models.

When training an image model, selecting training images that strike the right balance between consistency and variation is essential. It ensures that certain elements, such as perspective and scale, remain consistent across scenes while allowing for unique and diverse elements. I trained multiple models with different datasets before achieving the desired results.

Training data (generated from the standard model):

Generated samples from my custom model:

I built a preview server to visually inspect the generated images for quality and compare them to the scene descriptions. They all displayed the desired similarities while still being unique, ensuring a cohesive yet varied experience for the players.

Assembling the Game: Bringing AI-Generated Content Together

With the AI-generated scene descriptions and images in hand, I updated the game to fetch images dynamically. I built an asset server with a simple AI pipeline to fetch and serve scenes on demand. First, a request for a new scene definition was made to OpenAI's Endpoint using my fine-tuned model. The resulting scene description was then validated to ensure the presence of all necessary keys.

Once validated, the scene description was sent to Leonardo to generate an image using the custom image model. After receiving the generated image, I combined it with the scene description and sent it to the game.

The whole pipeline would take up to 20-30 seconds to run, which was too much latency when requesting a new scene, so I built a simple cache that maintained a baseline number of generated scenes to draw from. The cache would be filled up automatically in the background when reaching a minimum threshold. In this way, I was able to achieve an instant, endless and scalable supply of unique generated content.

The Result: A Unique Gaming Experience

The final product of this project offers players a unique and immersive experience that would not be possible without the use of AI. This innovative approach to game development satisfactorily demonstrates what is possible with AI technology. Yet this just scratches the surface.

Opportunities for Expansion

While the game I created provides a compelling experience, there are several opportunities for expansion and further improvement. For instance, the generated images are currently low resolution (512 pixels). By incorporating an AI upscaler into the pipeline, higher-resolution images could be achieved, albeit at the cost of additional time and expense.

To further enhance the gaming experience, incorporating more user customization could be considered. For example, allowing players to select themes to guide the type of scenes they encounter. Or even generating scenes based on the user's time, location or current weather conditions for a truly immersive experience.

Furthermore, the process and techniques used in this project can be applied to other projects, opening up a world of possibilities for AI-generated content in various disciplines.

The game

You are lost and must find your way home. Explore the foreboding forest while managing your stats and remembering where you've been. Collect navigation items along the way to help. Enjoy the scenery, but don't get lost!

You can play it at https://enegames.itch.io/lost.

This game was intended as an exploration of the theme of a physical solo card game I am working on. I wanted to create a strong abience and try out some ideas that didn't work in the physical version. Unlike the card game, a core mechanic of this game is plotting your way through a mental maze. If you make the wrong step, you might run out of courage.

Initial design

I wanted to keep this game very simple. At first, I considered doing it in PICO-8, but I wanted to learn pygame and try out replit. However, I intended to keep the same 8-bit aesthetic of PICO-8 games.

I imagined a series of top-down screens that the player's character would move through, switching from one screen to the next when reaching an edge. This was very much inspired by an impressive PICO-8 game called Ascent.

For gameplay, I imagined each scene would have a central encounter that would impact your stats and you would have a choice about how to respond or use resources that you collected to help.

I really liked the idea of a Fog Of War effect to limit what you could see until you were right on top of it, and I particularly liked the idea of linking the size of it to your courage stat. Similarly, I wanted your vigor stat to impact how quickly you could move through each scene.

With all of that in mind, I turned to my AI tools, which immediately changed the aesthetic I had pictured.

The process

Assets

I used Scenario with the intention of training my own model to get consistent custom backgrounds for each screen. I didn't have any graphics to start training with, so I got started by playing around with single-image generation. The prompt I used was:

pixel art, retro, gloomy, minimalist, dark lighting, dramatic lighting, top-down, 8-bit, game asset, tiles, 4-way intersection of paths meeting in a dark forest

Now my prompt might not have been great, but the AI didn't get the top-down 8-bit style I wanted at all. I realized after a bit that it was because the sample pixel art model I had chosen was strongly trained for a specific look. I got a bunch of nice, but unuseful, images that looked like this:

I experimented with some different models and prompts and started to see some very cool potential images with a dream-like quality that worked great. The problem was that my favorite ones had an isomorphic diorama style despite my best prompt attempts:

8-bit, pixel art, dark lighting, top down, game tile, dark forest, paths forming a cross, gloomy, mysterious, digital art

I decided to roll with it and changed not only the aesthetic but also the gameplay from a 3rd person avatar to a quasi 1st person POV. This became a theme: letting the AI guide the direction of the game. The AI had set a gloomy, glowy, muted yellows and green color pallet that I came to love.

After that, I made a ton more similar images and used them to train a model, and made a ton more from that, both unguided and with specific scenes in mind. Overall a lot of good images came out, but it was very hard to get the specific scenes that I asked for and I probably needed to do a better job with the training data. I experimented with different styles, some working better than others:

I also tried out Leonardo.ai which gave me some initially better-looking images, albeit without the nice dreamy look from Scenario. I tried training a Scenario model with those images, but it still was too overfit. I also tried Leonardo's custom models which were also hard to control.

I worried about the images where the isomorphic tile didn't fill the frame. I manually used Leanoardo's in-painting in these cases and was pleased with the results.

In the end, I grabbed a handful of images from both platforms that would work in the game and started coding.

Code

I hadn't used pygame before, so I relied on Replit's AI, called GhostWriter, to scaffold the main game loop for me. The first thing I wanted to see was how well the images would work with the new perspective, especially the movement. I asked the AI to load a few of my generated background images and move them around with the arrow keys.

Getting immediate feedback between code changes and Replit's browser-based output tab was very helpful. Overall, the effect was good, but I wanted it to be more immersive. I had an idea on how to cheaply fake out depth in the 2D image by zooming in when moving up and vice versa. Here are the results:

The image on the left is just panning, while the right image has the described effect. While subtle, I think it makes a huge difference. It also caused a lot of problems, not only in challenges with the code but also in limiting the images that it worked well for.

Code wrangling

The experience of leaning on AI to write the code was mixed between moments of joy and moments of frustration. Certain things worked out very well while others didn't work at all. I think game dev is especially hard for AI since a lot of it is playing with equations to get the feel right or debugging performance issues. Many times I had to rely on my own knowledge of game dev patterns to get things working right.

For example, I spent a lot of time figuring out the unnecessarily complex AI-generated code for moving the backgrounds around to figure out how to know when reaching the edge of a screen. Even worse, the frame rate dropped to single digits when moving up and I had to rewrite the background moving and scaling code to find the culprit. I wish the AI could have caught it for me, but it would identify potential issues that weren't actually the problem.

On the positive side, the AI wrote some useful code on the first go for several parts. For example, I had the classic issue where moving diagonally moved faster than moving just up/down left/right (because of Pathgoreum's Theorem the speed is 1.4 times as fast). I told the AI to rewrite the movement code to use normalized vectors for consistent speed, which it did just fine.

Another example is even more interesting because not only did the AI design the code, but also the visible output. I explained to the AI that I wanted a grid for the map with dots to show where you have been. It wrote a bunch of code that looked right, but the fun part was running it to see how it looked. It looked great, and I left it exactly how the AI designed it. The same thing happened with the stat bars. I originally pictured them looking much different, but I let the AI code it and checked how it looked and liked it better than what I had in mind.

The last example highlights that theme again of treating the AI as a co-creator and not simply a tool as a means to an end.

Content

I wanted each screen to have a little blurb and an effect on your stats. I designed a manifest file with the data for each scene that I had, 16 in all to fill the 4 by 4 map grid. I wrote out the declarative scene object as follows:

{
  "filename": "bear.jpg",
  "description": "A massive bear stands tall, its powerful presence filling you with both awe and fear. You tread cautiously, knowing any sudden movement could provoke its fierce wrath. Best not to linger. \n(Courage -2)",
  "stat": "courage",
  "diff": -2
}

I then gave it a list of the 15 other background filenames and told it to follow the example and it did. Some of the wording was a bit flowery so I told it to adjust the tone. I also told it that I wanted to have a separate description and stat effect upon discovery and return, and it updated each scene accordingly, adding extra structure to each scene object. In the end, I had 280 lines describing what to say and do for every scene.

Next steps

I spent a little time exploring AI-generated music. SoundDraw was an impressive tool and I got a soundtrack that felt perfect, but I clicked away and lost it, never to be heard again, which is both sad and beautiful. I think real-time generated music per scene would be more interesting, but I didn't explore that yet.

On that note, generating each scene for every playthrough at playtime would be the ultimate extension of this project. Every playthrough would be unique, no scene would ever be seen twice across games. While the map shuffles each playthrough currently, an entirely new generated map would be much more interesting and more fun. An infinite maze to explore and survive.

Technically, I think this is very doable. I can use a finetuned LLM to spit out new scene manifest data on demand, then use the description from those to generate a background from the custom-trained image model with Scenario's API.

For me, exploring all of the generated images, most of which didn't get used, was my favorite part of making this game. I want to have that experience as part of the gameplay. That might very well be my next project.

I used Replit and their AI “pair programming” service called GhostWriter on a new game project recently. This is my experience.

I had three goals when leaning on the AI:

1. Instantly go from zero to something interesting on the screen.

2. Add features without ever referencing any docs, tutorials or Stack Overflow.

3. Avoid making coding decisions based on my own experience with game dev patterns and general coding patterns.

I should say that I am an experienced programmer, and have made games on multiple tech stacks before, however, this project was in a new stack for me, specifically Python and pygame. The game was relatively simple, involving 2D, top-down, fog-of-war style movement in a minimal design with heavy ambiance, with game play as a mix of stat and resource management and maze solving. (A write-up on the game will follow soon.)

I’ll only talk about the interactions with the AI and not the rest of what Replit offers, though I did enjoy the zero-setup and instant hosted feedback loop, which made iterating with the AI possible.

Move over, I’ll drive

At first, I wasn’t sure this process was better than my usual workflow. But after working with it more, I decided it was. Although not everything worked as well as I had hoped, I got to a functioning game without the lead time I usually need to put in learning new tools.

Plus it was just fun, simply asking for what I wanted to happen and having it appear. I would sum up the while experience by saying that it felt like the Top Gun quote:

“Don’t think, just do.”

Replit offers 4 AI tools: “generate”, “transform”, “explain”, and “chat.” I found “chat” to work the best and used that 99% of the time. I liked how “generate” was more embedded in the editor, but it didn’t seem to have the context of the rest of the code and would spit out generic coding solutions completely unrelated to pygame. The same was true for “transform” and even “explain” which is a shame because “transform” seemed like a power tool. The best part about “chat” was how it knew about the different files and classes I had and could make cross-cutting suggestions, refactors and explanations.

That said, using “chat” all of the time meant a lot of copying and pasting. I wish there were a feature that would show a side-by-side diff of the proposed changes for you to review and accept. I believe Copilot X has this feature.

I found ways of making “chat” a little more useful, such as asking for only the parts of the code that changed, along with line numbers. I also got better results when prompting the AI to “act like an expert and use best practices.” For example, with that prompt, it corrected an earlier suggestion that used a deprecated pygame API.

I also found it useful to start with a basic request for what I wanted, then another iteration to refactor it, then another to enhance it closer to what I actually wanted or to fix what didn’t work. Perhaps some better prompting (like the “think in steps” trick) would have had better results, but I ended up doing the steps for it.

To that end, I think that if I did not already know a good amount of programming and game dev that this would have been much harder. Even just simple things like asking to “Extract this value out into a constant” or “Make a class for this part of it.” But I also had to draw on my game dev knowledge, like asking for alternate ways to do collisions or use sprite groups, and frequently, “Can you do that using vector addition instead?”

From https://openclassrooms.com/en/courses/4554386-enhance-an-existing-app-using-test-driven-development/5095731-understand-the-red-green-refactor-methodology

Basically, the AI was missing the “refactor” step in the Extreme Programming idiom, “red, green, refactor.” This was one area where working with an AI felt very different from working with a human pair. A human pair would be challenging the decisions I made, while the AI blissfully did whatever I asked for. Maybe this is an opportunity to improve the UX. For now, I just periodically ask the AI to do a pass over the code looking for improvements or potential issues, but it still feels like a one-way street. I would love the AI to say to me of its own volition, “Hey Jeff, the code down here looks like it should be changed…”

On a similar note, I wasn’t doing Test Drive Development for this project, at least not in the traditional sense. As a game, most of the “testing” was moving around in the game and seeing if things moved correctly, clipped at the right spot, had the right speed, weren’t jumpy, etc. Traditional TDD doesn’t help much with that. So after every code change, I’d have to manually see if it “felt” right and explain the problem to the AI if it didn’t. This was not a good feedback loop, as the AI would say that it “fixed” the problem, usually by putting more math around something, like ‘alpha = int(255 * (1 — y / radius))’ and it wouldn’t get the desired effect. I think a more traditional and observable testing loop would have worked better, and I also think that a domain that didn’t involve math and “feel” would help the AI shine more. But we got there, often by me puzzling through the math-y bits to make it look and feel right.

So where does that leave me? I am pleased with the state of my game so far, but I am slightly worried that my code isn’t idiomatic, doesn’t follow best practices, or most distressing, isn’t performant. Where does code linting fit into the pipeline? How would it be if the project got huge? Large code bases are always difficult to keep in check, but would that be better or worse for the AI to deal with?

Then, the deep questions emerge. Do humans ultimately even need to see the code? What if the AI controlled the code entirely, invisibly updating it based on the conversation with the human? We aren’t there yet, but it seems like we could be soon. Is that desired? Or would that set us up for problems in the future? The movies always have a manual override that the protagonist needs to activate (usually in the most precarious spot). What if the code gets to a point we can’t access or understand if things go terribly wrong? GitHub’s Copilot guides emphasize that the human is the pilot, and the AI is the assistant. I think that is good advice, but with things advancing so quickly, I wonder how long it will hold.

A human pair would be challenging the decisions I made, while the AI blissfully did whatever I asked for

I started with some goals. Did I meet them? Not completely. I certainly was able to quickly make things happen without having to know anything. I continued to feel empowered to blaze forward, adding features, though ultimately I lost most of the time I saved by tinkering with the math-y bugs.

I’m sad to admit that I did turn to the docs and even Stack Overflow, though often those instances verified what the AI had suggested but I had initially doubted. I did find it very helpful to be able to look up what I needed in the docs, but being able to go in with existing code gave me a much better context to understand and apply what I looked up, and I was able to improve what I asked the AI for.

The biggest difference in expectations was how much I drew on my own experience throughout the process. While I think that AI is fantastic for coders just starting out, it can be a very powerful tool for programmers with plenty of experience under their belts, and it becomes less of a magical code genie, and more of a magic wand to wield to shape your domain.

The final question is: will AI coding assistants change the way I work from now on?

The answer is absolutely yes. Since this experiment, I’ve been using AI in multiple effective ways professionally, and I plan to continue using it personally too in every new project going forward. If you haven’t tried an AI coding pair yet, I recommend giving it a shot, and if you have similar or different experiences, I’m curious to hear about them.

(Originally posted at https://medium.com/@jschomay/adding-inworld-ai-characters-to-an-existing-game-part-4-going-deeper-2c9f4f1084c3)

Inworld lets you make really interesting AI characters. But interesting AI characters don’t automatically make good game NPCs as I found out last time. How can you use AI to enhance your game in a way that only AI can?

There is a whole exploration of agentic AI NPCs that can understand and manipulate their world and other NPCs around them, and grow and learn. As cool as that is, this article focuses on making conversational AI interactions as best as possible.

Jon Radoff has a great article where he explores his list of “42 elements of fun” in game design through the lens of generative AI.

I found this article very inspirational, and wanted to explore some of its concepts more thoroughly, and how I am using them with specific examples from my game (you can play the demo here).

Exploring a World

A big part of the fun of playing a game is exploring a new world that is different than your own. World building can be a huge undertaking, but the depth of lore is what makes a game world feel solid and alive. Generative AI promises unlimited content to explore at minimal expense. All the player has to do is ask. But this is not all good.

One common problem is hallucination, when characters make up things that aren’t true or don’t make sense. For example, Carl the coffee cart vendor once gave detailed instructions on how to get from one subway station to another. The problem was that the lines and stations he mentioned do not exist in my fictional subway map! There are measures you can take to avoid this, but it is always a challenge and offers no guarantees. But that is only one problem.

“The subway system was built by a bunch of different companies and organizations over the years, but the most famous is probably the Interborough Rapid Transit Company, or IRT for short. They were responsible for building the first line, which opened in 1904.” — according to Carl, the coffee cart owner

The bigger problem is ludo narrative dissonance. This is what happens when what is going on in the story does not align with what is going on in the game. For example, my protagonist has no interest in the history of the subway, or the politics involved in its construction, or the technical aspects of its maintenance. He just wants to get his promotion at work. And yet, AI NPCs like Carl open the door for a player to literally talk for hours about those topics if they wish. While that type of content might be interesting to the player, it pulls them out of the primary game experience.

I have found that a good way to deal with this problem is to load up the characters with in-game knowledge and prompts like rumors about a suspicious person at a specific station, or potential train delays. This kind of content would make the player curious about what they are foreshadowing, and provoke the player to move in that direction. It adds tension to the game, and gently guides the non-linear narrative, while still allowing the AIs to riff.

I also found that putting interesting things in the game world for players to ask characters about is a good way to make room for lore for players to dig into at their own pace. For example, the broken vending machine has a backstory that foreshadows an important character that arrives later. But I tried to maintain the forward momentum by training Carl to warn the player that they will miss their train if a conversation goes on for too long.

Personalized content

The best thing about AI NPCs is that you can say anything you want to them, and they will say something meaningful back. This means every player’s experience will be unique, and they will have played a role in defining their own experience, which makes it personal.

A lot of this comes automatically from how Inworld works, especially how they handle long term memory and shared knowledge. But you could push it even further with custom game hooks that tie into Inworld’s scene triggers or character goals. I’m looking forward to trying their upcoming “intent” feature which should allow for less rigid content hooks.

Humans love sharing a good story, especially about something that only happened to them. I had something interesting happen in one of my recent plays, and I’m dying to share it:

Me (to Snarky Coffee Girl):
I have to talk to someone. My boss is super mad at me.

Snarky Coffee Girl:
Oh, I’m sure your boss is thrilled with your work ethic. Maybe you should buy them a cup of coffee to make up for it. Just kidding, sorry to hear that. Good luck!

Me:
Actually, that’s a good idea, give me your fanciest drink.

Snarky Coffee Girl:
Oh, my fanciest drink? That would be a cup of hot water with a tea bag on the side. Enjoy!

Me:
Come on, I’m trying to make a good impression here.

Snarky Coffee Girl:
Oh, I’m sorry, I didn’t realize your boss was a tea connoisseur. In that case, how about a vanilla latte with extra foam and a sprinkle of cinnamon? That’ll be $5.50.

Me:
Yes, do it!

Snarky Coffee Girl:
One vanilla latte with extra foam and a sprinkle of cinnamon, coming right up! That’ll be $5.50.

Me:
Oh shoot, I gave my last change to the bum. All I have is 5 bucks.

Snarky Coffee Girl:
Oh, well isn’t that wonderful. Looks like you’ll have to go beg the bum for some change now. Have a great day!

Me (to bum):
I’m sorry man, um, this is awkward, but um, can I have some change? I need 50 cents to pay for my coffee to bring to my boss so he won’t be mad at me.

Bum:
Oh, mmmmbbb sure, I can give you some change. I know the feeling of needing to please the boss. Here you go, man. Good luck with that coffee!

Me:
Wow, that’s awesome of you! Thanks so much.

Bum:
Sssbbble no prmmmmmlem, man. Jjjjjust be careful down here, the rats…they’re everywhere. And the tunnels, you gotta watch out for them. Mmmbbbbye now.

Me (to Snarky Coffee Girl):
Guess what? I asked the bum for change, and he gave me fifty cents. Now I can pay for that coffee. What a great guy.

Snarky Coffee Girl:
Wow, what a heartwarming story. I’m sure that 50 cents will make all the difference in your life. Here’s your vanilla latte with extra foam and sprinkle of cinnamon. Enjoy it while it lasts!

That scene may or not mean much to you, but I thought it was a great little vignette, and as the game designer, it was awesome to help create some of the story by playing the game instead of making it! (I’m also pleased with how the bum managed to mumble “no problem,” it was hard to train that.) I’ve mentioned before how AI characters are masters of improv, and this interaction definitely felt like I was a part of an improv skit. I mostly led first part, but the scene turns thanks to Snarky Coffee Girl’s brilliant insight of comedic reversal in asking a bum for change.

The only downside of this kind of interaction is that it happens in a separate layer from the rest of the game engine. For example, I don’t get 50 cents or a fancy coffee in my inventory to actually give to my boss. But I think that is alright. With some extra hard work, and taking some guesses at anticipating player’s actions, it might be possible to really push the envelope by using the Inworld intent system, their game trigger id system to feedback into my rules engine, and some extra generalized rules to make an in-game response to a chat-initiated action. While only a few players would likely experience the fruits of that labor, it would really knock their socks off!

“Hidden” content

Many games have achievements for finding hidden content, and mine is no exception. For example, there is a hidden way to get a soda from the broken vending machine. But it would be interesting to make a whole meta-game of hidden topics that only the AI NPCs know about, especially if they thematically support the game or tie in later. The player never knows if they will hit on a topic that goes deep. This was one of the best parts of classic parser interactive fiction games.

I have started doing this by giving characters knowledge of hidden tunnels, ghost stations, and a secret underground cult. All of these become game mechanics much later in the game, but make for interesting speculation from the first scene forward. You can see the bum above mentions it, but he also mentions a lot of other crazy conspiracies. But if you ask Carl about them, he will corroborate the rumors, though he doesn’t believe them.

Nurturing

One of the obvious applications of AI NPCs is dating sims. While my game doesn’t have a romantic side plot, it does have a few characters that the protagonist must successfully “court” in order to gain access to the next area. These are usually very static in my scripted content, so this would be a great place to let the AI be the gatekeeper.

For example, at one point the protagonist gets in trouble and has to talk his way out of custody. That conversation could be a good micro relationship arc and would be a great chance to use AI. Other characters show up at different parts of the story, giving the player an opportunity to build up a relationship over time. These relationships could be solely thematic, or they could influence certain game mechanics later. For example, the Girl in Yellow is a mysterious character that disappears when you try to interact with her, but later she becomes an important gatekeeper. It would be fantastic if the things you said each time you see her come up again when she finally confronts you.

Ghost station?

Commentary

I’ve always found it interesting how large of a role commentators play in sports. Since I first started thinking about adding AI to my game, I wanted to explore the idea of a commentator who sees what the player is doing and comments on how “well” the player is playing the game, or not. While this might have too much of a “Stanley Parable” vibe, this could manifest as the protagonist thinking to himself in between interactions about what has happened so far, and even what he thinks he ought to do next. I love the idea of making the Player Character also be an NPC!

All together now

As I am writing this article, I am realizing that many of these techniques overlap and work well together. As I begin to incorporate them into my game it feels like the process has shifted from world building and character design back to integrated game design, which feels right. I have a lot more to do, and a lot more to discover about getting the most out of my new AI cast, but this is a great start.

You can play the in-progress demo at https://deadline-ai-server.fly.dev/.

What I learned from my players and my characters

“Giving infinite dialog agency to the player in a controlled story is a frightening thing!”

Last week I released a demo of the new AI characters in my narrative game. One of my favorite things to do now is reading the (anonymized) transcripts from players interacting with these AI characters.

In my previous article, I had left feeling lukewarm about this experiment. Specifically, I was concerned with the free-form AI chat detracting from the tightly scripted main narrative and game experience. However, I’ve had a chance to reflect and get feedback, and now I’m feeling more enthusiastic. In this bonus developer diary I want to share what I learned and how my view changed.

In one interaction, the player asked Carl (the friendly subway station coffee vendor) if he had seen anything strange going on. Carl said he hadn’t lately, but one time he saw a pigeon fly into the subway and cause a bunch of chaos! What a lovely little anecdote for the player.

More importantly, it is one I never would have written myself. And Carl probably won’t share it with any other players again. It was unique for that particular player, and even better, that player collaborated in telling the story, making it more personal and meaningful. That kind of interaction is an elusive holy grail in content-driven game design. And that is the true promise and power that generative AI offers.

In another case, a more devious player was having fun trying to “break” Carl. Despite his protests, they eventually managed to coerce him into serving them 100 cups of coffee! Then, they faked a heart attack and poor Carl had to improvise in the subway setting to seek emergency assistance (which he did rather well). It was quite tense for a bit, and the player and Carl managed to laugh about afterwards.

Only one player was directly inappropriate. I was so proud of Carl, who, instead of the default AI safety-control response (usually like, “I’m uncomfortable talking about that, let’s talk about something else”), managed to stay in-character to defuse and redirect the conversation, even across multiple uncomfortable interactions.

I never tire of watching Snarky Coffee Girl relentlessly troll players, no matter how hard they try to outdo her. Once, she slipped up in some kind of AI safety over-correction bug that made her announce that she was a “virtual assistant.” When the player asked if she really just admitted to being a virtual assistant, she just rolled with it, saying “No, I’m a figment of your imagination. Of course, I’m a virtual assistant. Do try to keep up, will ya?” I was laughing out loud, and I imagine the player was too.

I had such a good time reading the transcripts, that I was reassured the players must have been enjoying the experience just as much. These example interactions, and many others, showed me that AI characters are masters of improv. Watching my characters, out in the wild, managing on their own, taught me an important lesson.

Trust

Although my characters sometimes faltered, or said the wrong things, or spoke nonsense, for the most part they did just fine, and frequently pleasantly surprised me. As a game designer in this new world of generative AI, my job is to design the characters as well as possible, and then trust that they will help carry the game.

It’s almost impossible not to anthropomorphize my characters. I know that they are nothing more then LLMs and prompt engineering, and each interaction is running binary data through a model to complete a string of text word by word. But none of that really matters. Functionally, I need to trust them, in the very same way I would trust trained improv actors that I hire to entertain an audience.

I also learned to trust the players. At first I didn’t know what players would say to my characters. Now I do. Most played the part of the protagonist character, conversing about what they have seen and experienced so far. Many simply asked, “How’s it going?” A few were more interested in peering behind the curtain and exploring the mechanical workings and limits.

In the end, I realized that it didn’t matter what players said. The players that role-played likely had a great time. The players that probed the dark corners, trying to break things, probably had a good time too. Ultimately, with or without AI, the game happens, not on the screen, and not in the script, but in the player’s mind, and they will enjoy it however they want.

Prompting… the player

While I am pleased that players are enjoying their experience, I still want to use the AI characters as optimally as possible. I have some directions to pursue to achieve that.

This whole time I’ve been concerned with trying to make better AI character prompts. And while that is important, it is equally, or even more important to prompt the player. Everything in a game is a cue, a little nudge, to spark the player’s curiosity, to go check it out, and to progress in the direction you want them to go. So how do you prompt the player?

One way is with rich content. In a primarily text based game like mine, the static content creates a framework of context for the player to reference. In my case, this includes all written text, but also, everything available to discover in a location, like my broken vending machine. I was somewhat surprised to find quite a few players asking Carl about it. Even though I didn’t design Carl to know that the vending machine was there, the context of the conversation was sufficient for him to reply along the lines of how the subway maintenance has been falling behind. This is perfect, because it foreshadows everything that is about to happen to the player. (The broken vending machine is a metaphor? Wow! Thanks, Carl!) Which gets to the next point.

The second important place to prompt the player is directly in the conversations they have with the AI characters. Their responses should make the player even more curious, and even more eager to press forward. But, how do you do that when you don’t directly control what the characters will say. How do you trust your characters?

Giving the tools to thrive

It is a very strange thing to trust an algorithm. I trust Google Maps, and to some extent, self driving cars, but this is different. This is a facsimile of a living, thinking, intelligent entity. You have no way of knowing how it might react to a given situation, yet its response is so important. How do you trust something that is, by design, nondeterministic?

How to trust your characters

I found that the best AI characters came from the most interesting characters to begin with. Snarky Coffee Girl has minimal design behind her, but she plays her role superbly. I tried to make another commuter in the station. She is pretty bland, and serves more to show the player that the protagonist prefers to keep to himself. When I tried to add AI for her, she was really generic and even annoying. So it goes to show how even minor characters should be interesting. This is true in game design as well as in TV and film. As beloved screenwriting guru Blake Snyder advises, “give every character a limp and an eye patch.”

Once you have an interesting character, Inworld offers a whole suite of tools to direct how they respond. I’m still figuring out how to best use it all. Beyond turning all of the dials and knobs to get the right personality and tone of voice, I’m starting to learn that orienting each character’s place in the game is critical. For each character, ask why they are in the game and what they bring to it. You can probably tell that I already had a good answer to these questions when I initially designed the scripted interactions and narrative. Now, I’m learning to pull that into the Inworld system too. For example, I tweaked Carl to warn lingering players not to miss their train, and to encourage them to stand up for themselves if they mentioned their boss. As more events start to unfold, other characters should have opportunities to worry about (and therefore talk about) very in-game concerns. In some cases, this might require carving back some of the scripted narrative to leave some unanswered questions for the player to ask about.

Of course, your characters can’t talk about in-game things if they don’t know about them. I mentioned the broken vending machine earlier, and how Carl didn’t know about it. I plan to go back and add everything observable in the scene to the characters’ knowledge. It reminds me of how you would have to add all of the nouns in old school interactive fiction games to the world model to avoid embarrassing errors when the player asks about something they read about, but the author forgot to define it. Luckily, generative AI will just spout out what ever comes naturally. It’s still surprising to me to never see a character say, “Uh, I don’t know about that.” This trait is pretty sweet, but you lose control of that opportunity to prompt the player if the character doesn’t have the appropriate context. Luckily, Inworld offers a number of ways to make your characters aware of things.

For each character, you can define common knowledge, private knowledge, and scene events. I’m still working out what combination of these work best, but the idea is that character conversations will be more rooted in the game world. For example, I look forward to seeing conversations with Carl where he brings up fun anecdotes about the broken vending machine in the corner. If the player hasn’t checked it out yet, they likely will after Carl mentions it. Better yet, maybe I hint to Carl that it broke because the Skater Dude was trying to steal from it. The protagonist character mentions earlier how much he hates rule-breakers like Skater Dude (who plays a big role later in the game), so this would be great contextual fodder for Carl to draw from.

And the rest is trust.

Next

In the next article, I’ll get back to the scheduled plan of more deeply exploring ways to successfully integrate AI characters into games. And in the meantime, I’ll be improving and adding characters to my game!

(Originally published at https://medium.com/@jschomay/adding-inworld-ai-characters-to-an-existing-game-part-3-integration-and-analysis-826172445e10)

My game now has live AI characters in it. Interacting with these characters directly in the game is novel, fun, and even surprising.

But I can’t help feeling like something is lacking. Do the AI characters add to the game, or detract from it? Or are they simply inconsequential?

In this part of the series on adding AI to an existing game, I share the experience of integrating Inworld characters into my tech stack, and I consider if the result was a success or not, and if it could be better.

Getting trolled by own characters (“Snarky coffee girl” in this case).

Getting it all connected

I have already created a few Inworld characters for my game. And I have the existing game code. In this case, the game is a browser-based game, so all of the code is in JavaScript.

Using the Inworld SDK

I followed the Inworld docs for the web SDK. These required setting up an additional node.js server to securely create session tokens. Other than that, the basic process involved:

• Connecting to the Inworld server

• Activating the correct scene or character

• Sending out messages

• Receiving the responses

I had a little confusion figuring out how to correctly use scenes, and how they effect switching between characters, but it wasn’t too hard, and the Inworld Discord support channel is very active and helpful.

Although the SDK includes audio chat and character emotion states, I did not use those advanced features since this game is mostly text based and I wanted to keep that interface.

Integrating with the game engine

As for integrating with my custom engine, I was easily able to add the Inworld character IDs into my engine’s stateful “manifest” of game entities. Not only did this make it simple to “opt-in” to adding AI interactions to some NPCs and not others, it also made it possible to enable or disable this feature on a per character basis at different points in the story.

It also made it simple to swap out IDs during game play as well, which would be helpful for connecting to different “versions” of the same character with different configurations. Or, in my case, the same game entity, “Coffee cart,” has different baristas at different times, so I could swap IDs for these different characters without needing a unique coffee cart entity for each barista.

Inworld has a concept of character goal triggers, and scene triggers. I used these in an attempt to make the characters change how they related to me as the story progressed. I got a less dramatic result than I desired, but I think it could be improved by tweaking my character prompts in Inworld’s studio.

Extending the UI

I was able to reuse the existing game views for the additional AI interactions, which not only made integration simpler, but also kept a holistic feeling to the interactions, which was one of my original goals. Both the chat interface and AI responses appear in the same UI as all other narrative content, helping the chat feel like part of the main story.

I had initially considered a few approaches on where to add AI interaction. I went with the option of adding a “Talk more…” button after the scripted content finished, which enters a free-form chat-like interface with AI generated responses.

This option is only available on certain characters and at certain times that fit with the story. For example, the protagonist mentions that he doesn’t like to talk to the other morning commuters, so interacting with one of them doesn’t offer the “Talk more” option.

But is it better?

One of my main goals in this experiment was to have the addition of AI make the game and story a better experience without detracting from the original curated content.

Ultimately, the goal is to make the world feel more alive, but also more interesting and deep, and to add tension to the story.

The characters did indeed feel more “alive.” I must admit that I felt excitement from the unexpected responses in some of my conversations, like when the “snarky coffee girl” trolled me, no matter how hard I tried to get into her good graces.

I definitely felt compelled to forget about the rest of the game for a while, and just keep talking with snarky coffee girl. But I wonder if the novelty would wear off. And worse, I worry that talking with AI characters can become a meta-game that distracts from the actual game experience.

Or perhaps, stopping to have a chat is part of the game play. I think this depends greatly on the type of game, and might not work well in my existing game that already has a strong narrative.

This is why I am left with a nagging feeling that I haven’t reached the desired experience. I don’t think this is a shortcoming of Inworld at all. But I think that in this case, I’ll have to get more clever with how I use their tooling to support that narrative that already exists in my game.

I need to craft my AI characters better, to make their responses prompt the player to feel the emotions and motivations that push the game and story forward.

When done well, I think AI characters can do this more successfully than scripted characters since they are, by design, reactive to a player’s personal experience.

Play-testing

Part of the challenge is that I do not really know what players will say to my characters. When I played, I role-played the protagonist character very rigidly, and measured the character responses against the story as I know it is supposed to unfold.

But other players will interact very differently. I need to play-test to see what conversations happen, and design around that. It may be possible that I’ve included AI in the wrong places, or maybe free-chat is the wrong mechanic. For example, it might be better to send background prompts based on the player’s actions and world state and then surfacing the character responses within the scripted narrative some how.

I invite you try out the demo. At this time, only a few characters in the intro have AI.

Play-test “Deadline” with the initial AI characters

If you have feedback, you can use this post’s comments or tweet me. Let me know what you think of the AI interactions, and if you’d like to see more of them, or see them work differently. Just a note, I’ll be able to see the content of your chats, but no identifying information.

What’s next?

In the next post I will dig deeper into exploring ways that AI can enhance game-play beyond just chat bots, and how I might apply that to this game.

After considering some implications of adding AI NPC’s to an existing game in my previous article, I have taken the first timid steps into creating my first character in Inworld. I want to share the experience, because it was strangely visceral, magical, and unexpected at the same time and made me think things I didn’t realize would come up.

This isn’t my first foray into AI character creation. I’ve played around at a low level with building templated prompts by hand to feed to GPT3 in order to get a rich and meaningful narration of a character action from player input. But Inworld has a very nice suite of tools for defining your character and its world that feels more like character building and world building than LLM prompt engineering.

I want to be clear that what I have done so far is extremely shallow, only the bare minimum of character creation, and the very beginning of what I expect will be a long road of tweaking and testing. Inworld keeps rolling out new advanced features that I can’t wait to dig into, but even at this early state, my experience felt worth sharing, so here it is.

Meet Carl

The game that I am adding AI characters to is called “Deadline.” It is a narrative-heavy game set in a fictional subway system where players help the protagonist, Steve, get to work on time to deliver a job-defining presentation, except that on this day, the subway is going haywire and Steve’s world is falling apart around him.

You can play the original version online if you’d like.

Carl’s first (scripted) words

In the very first scene, the player finds a small handful of characters and items to interact with, including the “Coffee cart” where Carl works. Carl is likely to be one of the first characters that players interact with.

Carl plays a very small role in the game, serving as a way to introduce players to the UI and basic mechanics. They learn that sometimes interacting with things give you inventory items (the ubiquitous to-go cup of hot coffee in this case). But more importantly, they get a very subtle introduction to Steve, the character they will be guiding around the subway for the next hour or so. They get a hint that Steve is a man of routine, and an idea of his financial position in the world, and that he likes order, rules, and efficiency.

Carl also introduces the plot and provides some foreshadowing. Over the course of the first few days that make up the introduction / tutorial scene of the game, Carl is a regular, and his energizing brew dispensing makes a natural conversation starter around Steve’s growing loss of sleep and mounting stress as he prepares for his high-stakes and overwhelming presentation levied by his arrogant boss. And then, (minor spoiler here), just when Steve needs Carl’s friendly encouragement the most, he’s gone, replaced by an irreverent teen who could care less about Steve’s troubles. If the player clicks on the coffee cart again after this unpleasant exchange, Steve just says “I miss Carl.” And hopefully, so does the player, but there’s no time to lament, because Carl’s absence is only the first indicator of everything that is about to go wrong.

“I miss Carl.”

Why Carl?

I chose Carl as the first character to get the Inworld treatment, for a number of reasons. First and foremost, I wanted to start simple. Carl is a simple character with a restricted personality and motivations (which I later realized might have actually made things trickier) and he goes away after the first scene.

Also, Carl is the perfect tutorial character to get the player used to talking to AI characters with limitless AI generated content, since he shows up immediately, and doesn’t risk impacting the plot by going off script too much (which again I realized later was a false assumption).

Also, as far as Inworld’s internal state is concerned, Carl isn’t exposed to much knowledge in the knowledge base, and isn’t affected by goals or scene triggers or the other more advanced Inworld options. That said, I do intend to use Carl to experiment with these options later, like making him aware of which day of the week it is, and become more concerned and sympathetic over Steve’s growing loss of sleep.

Meet Carl. Again.

After filling out all of the basic information that Inform uses to personalize characters, Carl 2.0 was ready for his first words. While the ultimate plan is to integrate a UI in the game to converse with AI characters seamlessly along side of the scripted content, for now I just used Inworld’s in-studio character testing chat window.

Carl is about to start talking for himself. Would he talk like how I imagined? Would I like him?

But before I hit that button, I experienced a surreal thrill of anticipation. I was about to actually talk with a character that had only lived in my head for years. I had written every line of dialog he has ever said up to this point. And now he is about to start talking for himself. Would he talk like how I imagined? Would I like him? It was thrilling. And also a little frightening.

And there was one more important question: what should be my own first words to him? In a display of spectacular timidness and unoriginal thinking, I went with the first words I wrote for Steve so many years ago:

Wow.

How was it? It was both good and bad, magical and frustrating. But the first thing I noticed hit me like a brick. Do you see it?

He didn’t say “Sure thing, Steve.” He said “Sure thing, Jeff!”

I’m Jeff. Obviously, Inworld knows my user name and uses templated variables to inject the player’s name in the dialog. But Carl doesn’t know that! All these years I have almost heard Carl’s voice, saying “Sure thing, Steve,” but now Carl is talking to me. Carl’s first words were directed to me.

Carl doesn’t live inside my game any more.

OK, once I got past that, the second line is interesting too, because it came directly out of the example dialog that I gave to Inworld. Very in character, but not original. But then, Carl immediately overcompensated:

Carl instantly getting too creative with his new found freedom

Here’s the thing, the train schedule most certainly did not change, and worse, later on the player learns that Steve’s train actually leaves at 6:30 AM and a ticking clock becomes a central piece of the plot as Steve races to get to work on time to give the presentation.

AI hallucinations like this can be nice for giving flavor and depth to the world, but detrimental in strongly plotted context. This is one of the biggest challenges in introducing generated content. In my earlier article I brought up this concern. I know that Inworld and the rest of the AI industry is working on issues like this, but for now we have to find ways to make it work.

I conversed a little more after that, but Carl got very chatty and, to my ear, sounded very “on the nose,” which is something I feel I notice in a lot of AI generated content. He also spouted some more untruths, saying, “This is the busiest station around,” when it actually is designed to be one of the more quiet stations away from the larger downtown ones that the player will arrive at next.

Tweak Time

While I wasn’t pleased with how “Carl” turned out, I was very aware that the real work was just beginning, the work of tweaking Carl’s textual DNA. Despite how far generative AI technology has come recently, we still are at a point where changing one word in a prompt can have drastic results, which is true in LLMs like Inworld’s models as well as other media like the contortions resorted to in Midjourney prompts (like this mouthful: “trending on artstation, high detail, high definition, high quality, photo-realistic, ultra realistic, hyper-realistic, 4k”).

For the second attempt, I removed the “knowledge of the train schedule” fact that I had first put in, and I changed a few of the personality and motivation fields to make Carl more terse and more likely to ask questions or engage in short small talk. There’s more to do, but the results are already better, and AI Carl sounds more like good ol’ Carl, though I’m still not sure what to say to him, or what players will want to say.

I have to admit, I didn’t quite get the joke, but I was amused nonetheless. This dialog mostly feels consistent with the picture I’ve had of Carl in my head all along.

Takeaways and Next Steps

I really enjoyed this experience of bringing my first character to life and hearing his first words. For me, the whole thing felt like as if you’ve ever imagined an upcoming conversation you were going to have, and played it over and over in your head, but then when it really happened it went nothing like what you imagined.

Adding AI to an existing character feels like learning to let go of a child, and having faith they’ll do well on their own.

Carl is just the beginning. I still have to integrate him into the game’s UI and experiment with the more advanced options. And then it will be time to pick a more challenging character, and go through this whole process again. To meet my other characters, again, for the first time.

As a game developer and narrative designer, I’m always looking for ways to make my games more engaging and immersive. Recently, I had an idea to take one of my old narrative games and add AI NPC characters to it using Inworld. In this developer diary, I’ll walk through my planning and design process for this experiment.

This is the start of a series of articles that follow my journey. You can see all articles in the series at https://jschomay.hashnode.dev/series/inworld-npc-explorations.

What is Inworld?

Inworld is a platform that lets creators easily build AI-backed characters by providing a personality, goals, and a knowledge base. They offer integrations with game frameworks to add these characters to your game.

What is “Deadline?”

Two of my favorite movies are “Subway” and “Kontrol.” A while
back, my friend told me that I should make a video game like those films.
“Deadline” is that game.

Welcome to the subway

“Deadline” is a narrative-heavy game with minimal visuals, map-based navigation, a central story to explore with a few side plots, and many oddball denizens of the subterranean labyrinth. In that way, it is a little bit like “Kentucky Route Zero.”

The goal of this experiment is to make those oddball denizens into AI-controlled oddball denizens… without losing control over the plot!

Challenges of Adding AI into an Existing Game

Making a non-linear narrative game is already difficult enough, but adding AI to it has a number of additional complications to consider.

Generated vs. Scripted Content

In “Deadline” the player can see a list of NPC characters and items in each scene, and can select one to interact with, such as the “Coffee cart” in the opening scene.

Items and NPCs in the first station in “Deadline” for the player to interact with

This brings up a context-aware dialog box that usually includes a description of what happens and some dialog, both for the player’s character and the NPC.

Interacting with the “Coffee Cart” brings up this dialog box

With the power of generative AI, it would be nice to create AI characters that have enough information to allow the player ask the right questions to move forward in the game. This could make for a very unique and interesting experience, and puts the player more in the driver’s seat, by making them think harder about what to ask, similar to old-school command line text adventure games.

On the other hand, I have lots of fun scripted content that I’d like players to come across. Also, pure generated content makes it technically much harder to trigger narrative rules that change the world state or unlock more content.

I could imagine a hybrid approach where narrative-significant interactions are scripted, while innocuous interactions use generated content so players can ask their own questions. But giving the player free rein some of the time, but not all of the time might actually make the game feel more railroaded. Plus some of my favorite scripted dialog moments happen in the minor interactions.

Another idea is on every interaction to let the player choose between advancing the story via the scripted content, or having an unscripted side conversation with an NPC. But this approach may also break the flow of the game by essentially having two ways to play it.

Furthermore, with a hybrid approach I’d have to carefully make the AI generated content match the voice and tone of the scripted content, which might not always be easy.

UI

The UI will need to change to accommodate the approach taken above. I can imagine a few approaches:

• Placing an input field at the end of each dialog box’s content for the player to type in prompts if they want to dig deeper.

• Offering a choice between one (or more) truncated scripted options and an input field for the player to supply their own prompt, and changing what displays next based on which option they chose.

• Adding a “chat” icon next to each NPC listed in the scene to activate free-form chat, and also keeping the original UI of clicking on the NPC name to trigger the scripted content.

Of these, I like the first option best because it feels the least intrusive, doesn’t preclude scripted content, and allows the player to interact further in a more free-form manner at a moment when they have the most new information.

Consistency

If one NPC is AI-backed, do all NPCs need that functionality? Does it break the fourth wall too much if players wonder why they can interact with one NPC but not another?

Adding AI to every character gets expensive and hard to manage. It also seems undesirable in many cases. For example, in the first scene, the player can say a quick hello to another character waiting for the same train to arrive. If the character tries to interact with that character again, the player’s character says “I think one hello is enough when talking to complete strangers.” This sets some bounds to the game, but more importantly conveys information about the player character’s personality. Allowing the player to then enter free-form chat mode with this character would break the constructs of the story.

I am hoping that selectively applying AI will feel natural without causing AI vs non-AI characters stick out like sore thumbs. I expect the choice of UI could help smooth this inconsistency and the narrative design could also make the presence or absence of free-form chat intuitive.

Staying “on Script”

One of the challenges of adding AI characters, is making sure they stick to talking about what they should know. Ideally they will be able to drive the plot forward and add texture and even foreshadowing. But if they go off the rails (sorry, couldn’t pass up on the pun), it could cause a lot of confusion for the player. Also, the player might ask things that don’t make sense, either on purpose or by accident, so I need a good strategy for dealing with those cases.

Ultimately, the goal with adding AI characters is to make the world feel more alive, but also more interesting and deep, and to add tension to the story.

Inworld should be pretty good at keeping NPCs on script, but I have a few questions. For example, is there a way to change a character’s knowledge base over time? And if so, does each character “instance” across games have its own knowledge base? I need to explore this further, but I think each character might be a single entity that has a single source of knowledge, regardless of how many people are interacting with it. If that is the case, I might need to get clever with adding prompt prefixes to what the player writes to “direct” the character, though I’m not sure how well that would work. I think this is one of the big challenges game developers face when introducing AI that can be unpredictable.

Conclusions and Next Steps

This experiment presents a unique set of challenges and unknowns, but I’m excited to dive in and see what happens. I’ll do some more research and pick a small spot to start with, perhaps Carl the coffee cart guy in the first scene. In the next part, I’ll cover some of the more technical aspects of integrating Inworld AI characters into my existing game framework.

Disclaimer: ChatGPT helped turn my rough outline into prose, which I then edited further into this final form.