The problem is immediately visible from the diagram: if the LLM takes longer to respond than the duration of a game step (which was the case in my experiments), the LLM ends up skipping some steps of the game world. It makes sense to send the LLM the prompt corresponding to the situation after the previous response has been processed, the action has been taken, and the result has been observed. Otherwise, the LLM will not see the results of its work.
Limits
Almost immediately, I ran into the limits. There are the obvious limits specified in the pricing plan: requests per minute (RPM), requests per day (RPD), and tokens per minute (TPM). All these values are calculated using a sliding window, rather than being reset once a minute/day.To avoid exceeding the per-minute request limits, I simply divided 60 by the limit value, and if the response from the LLM arrived earlier than that, I would pause the thread for the remaining time. On the free tier, lite models are allowed to process one request every 4 seconds (for comfortable gameplay, I set the server discretization time to 0.45 seconds, meaning that, in the best case, I can process the LLM only every 10th game step). There is nothing we can do about daily limits; after 500 requests, we switch models. Gemini itself advised creating several accounts, but I didn’t bother with that, especially since doing so from a single IP is immediately detected. My token usage is very modest and is barely noticeable relative to the limit. However, there is another limit that wasn’t specified: in the Gemini API, you can obtain a response ID and feed it into the next request (essentially continuing the conversation), and there is also a limit on the length of such a chain. The error is called “Conversation ancestor chain exceeds maximum validation depth.” This limit may not be fixed and may instead depend on how full the context window is. From my observations, I reach this limit after roughly 100 iterations.
Prompt generation
When starting the LLM thread, I immediately send a static request explaining the general rules of the game, what items exist, what can be done, the format of the response, and so on. I called this request the MASTER-PROMPT. It is partially written by hand, and partially populated with data generated by the “crafting system”: a set of items, recipes (after all, we don’t want to spend time discovering them ourselves?), buildings, and so forth. When the LLM returns a response, we take its identifier and provide it in the next request, see…the figure below.
Modern LLM APIs allow you to specify a response schema (in Gemini, it is called response_format). In effect, this is a strict requirement to return the response as valid JSON with the specified fields, some of which can be marked as “required.” You can specify the data type and additional properties for the field contents. This format greatly simplifies response parsing and eliminates pages of unnecessary text (and, probably, excessive token usage). I tried two approaches to getting a character’s action from an LLM:
For each possible action, I create a numeric field ranging from 0 to 1, and in the MASTER-PROMPT I ask the model to enter the probability of choosing that action in these fields. When parsing, I iterate over all the numeric fields and select the action with the highest value.
I create a text field called “action” and use an enum to list all possible options. The LLM then selects one of them.
I did not notice any particular effect on the result, so I chose the second option because it is easier to parse (and, again, should require fewer tokens). In addition to the action, I also make it return several fields: goal, plans, thoughts, and what to say. I make the last one optional, so the character can say something if they want to, or remain silent.
The prompt is generated from the received observations: what the character sees, hears, is currently doing, their inventory, the opponent’s inventory (if trading), and so on. Processing the observations allows me to determine the actions available to the player and add them to the schema. Incidentally, one of my earlier projects was useful for compiling the list of things that can be crafted.It also makes it possible to remove from crafting recipes those that consume resources needed for a more important goal.
Memory
Obviously, such “agents” need memory to function properly. I added “cell memory”—I remember important objects and their coordinates—and “event memory,” a record of previous actions. You regularly have to catch the LLM red-handed; it can calmly say “I’m putting copper, tin, coal, and a casting mold” into the smelter without having half of the listed items. For this reason, I added “deficit memory”: when I parse a line containing a list of items and find a mismatch, I record the missing item in this memory. You can then try filtering out actions that contradict the shortage. For example, if you have a coal shortage, you probably should not try to light the furnace or put ingredients into it. In other words, you have already tried before and found that there is no coal, so until you obtain some, there is no point in trying again.
Example of a generated promptYou stay at coordinates [0, 0] and facing East. You are staying, you HP = 50. Your inventory has changed. It is now as follows: bone(x133), stone(x298, equipped), piece of copper ore(x4), flint stone(x14), bone knife(x30), brick(x5), berry(x3), unfired brick(x16), unfired bowl(x1). You will need the following materials and tools to achieve your aim: clay (2 x), coal (3 x), tin_ore (1 x), handle (1 x), flammable (12 x), fuel (9 x). Pay attention to the places where they can be found or purchased. It probably doesn't make sense to build anything until all ingredients are found. You need to produce or buy these things to achieve your aim: bronze_pickaxe_head[1], copper[2], tin[1], pickaxe_mold[1], brick[10], bowl[1].You see: clay wall at[2, 1], wall at[2, 2], wall at[4, 0], clay wall at[4, -2], damaged granite wall at[3, 4], fire furnace at[2, 0], stone-like objects on the floor at[2, -1], stone-like objects on the floor at[1, 2], stone-like objects on the floor at[3, 0], stone-like objects on the floor at[3, -1], stone-like objects on the floor at[2, 3], stone-like objects on the floor at[3, -3], stone-like objects on the floor at[3, 4] and free cells at [2, -1], [1, -1], [1, 2], [3, 0], [3, -1], [3, -2], [2, -2], [2, 3], [4, -1], [4, -3], [3, -3], [5, -2], [0, 0], [0, 1], [1, 0], [1, 1]. The clay wall is in front of you. This could get you the resources you need. Mining the wall at coordinates [2, 2] may yield the resources you need. Mining the wall at coordinates [4, 0] may yield the resources you need. Mining the wall at coordinates [4, -2] may yield the resources you need. The fire furnace is in front of you. You can craft: bone_knife, bone_awl. Negative experience with the furnace: you put in 15 unfired bricks and 1 coal, ignite it, and the bricks remained unfired. Positive experience with the furnace: you put in 3 unfired bricks, 1 coal and 1 kindling, ignite it, and the bricks became fired.Recent events: timestep 291: the furnace fired up at [(2, 0)]; timestep 295: the furnace cooled down at [(2, 0)]; timestep 316: You cancel exchange/trade, there are no useful items to take or to give, don't trade/exchange with the same person/object; timestep 332: the furnace fired up at [(2, 0)]; timestep 336: the furnace cooled down at [(2, 0)]; timestep 358: You cancel exchange/trade, there are no useful items to take or to give, don't trade/exchange with the same person/object; timestep 374: the furnace fired up at [(2, 0)]; timestep 378: the furnace cooled down at [(2, 0)]; timestep 400: You cancel exchange/trade, there are no useful items to take or to give, don't trade/exchange with the same person/object; timestep 417: the furnace fired up at [(2, 0)]. Your previous actions were: use_flint_stone, mine, cancel, use_flint_stone, mine, cancel, use_flint_stone, mine, cancel, use_flint_stone. Your primary objective right now is to obtain piece of coal, bundle of kindling. Resolve this before moving on to the main goal. Your last aim was “Obtain piece of coal and bundle of kindling”, set 240 steps ago.
Results
In principle, it works, albeit rather crudely: the LLM-controlled player walks around, collects items, tries to extract metals from ore (and sometimes succeeds), builds structures, and once even managed to build a furnace, load it with the correct items, and cast a pickaxe head (one of the most complex recipes in the current crafting system). Before that, however, he wandered around for a very long time and found a pile of useful items belonging to some deceased dwarf, so many of the ingredients came easily to him.
At the same time, there is absolutely no way to call such an agent’s behavior sensible.He often gets stuck in some kind of dead loop: sometimes crafting the same not particularly useful item, sometimes spinning in place, and sometimes getting stuck in exchanges with objects. He doesn’t really understand coordinate descriptions and walks past useful resources; even when the prompt explicitly says—“on the right there is a wall with the minerals you need”—he walks right past it without turning.
Among the pleasant moments are good commentary, speaking to the point, commenting on his actions, and trying to talk to dwarf NPCs passing by.
A small example of movement in GIF format
Trading
One funny thing. The game has dwarves with a very good algorithmic AI. They purposefully extract resources, fight goblins, and know how to build, craft, and even trade. The only thing is that trading is implemented passively; they do not initiate exchanges with the player. However, they can evaluate goods according to their current state, refuse an exchange if the proposed deal is unprofitable, and, if the player asks, tell them what they want to buy.
My agent tried trading with the dwarves several times, but most of the time the dwarves refuse to accept the junk the agent offers. Nevertheless, he managed to make one exchange, though it was to his own detriment: he traded all the tin he had for 4 pieces of copper ore, 1 piece of coal, and 2 pieces of clay (that is, rare resources for much more common ones).
Fields in the JSON response responsible for the exchangeConclusion
The most banal and, at the same time, most devastating problem is speed.If the request were processed in half a second, it would be possible to do everything synchronously and, most importantly, process everything that happens in sequence without interruptions. Maybe we should try deploying some small local LLM.
Expectedly, LLMs are poor at spatial reasoning, and passing all topographic information as a list with coordinates does not work well.
We could also add conversational memory and observe how the agents communicate, but then Gemini’s limits definitely would not be enough, so we would need to look for an alternative.
At the moment, it feels like an LLM can be tasked with generating high-level commands and speech, but the implementation needs to be done using some more efficient methods. In other words, it can decide that it needs to approach a certain object, but how to do so would be better left to some other algorithm.
