Eventually, I ran into a choice: either keep constantly improving the bot's behavior by scripting the best behavioral options for it in every situation, or go down a completely different path.
If I had chosen the first option, I would simply have had to endlessly expand the scripts responsible for the bot's behavior algorithms, trying to account for all the main situations that could arise in my game. And there are thousands of such situations. I would also have to make sure that these algorithms did not conflict with one another.
Could I have gone down this path? Absolutely. Although it would not be particularly easy and would take a considerable amount of time, the option was entirely feasible.
However, I'm certain that in the end, it still would not be possible to script a bot whose battles would feel on the same level as PvP battles. The bot's limited choice of action algorithms, its templated actions in specific situations, and its overall predictability would not provide the enjoyment you get when facing a real person.
TAG39@@And not long ago, an idea came to me: why not try implementing a self-learning model in the game? One that would conduct thousands of battles against itself and learn to play at a level where competing against it would be genuinely interesting. What's stopping me? Nothing. So why not do it?At least, those were my grandiose plans. But it all turned out not to be so simple...
2. Starting conditions.
As I already said, I prefer to work on the project the old-fashioned way: I tinker with everything manually, configuring each element in the Unity engine myself.The only exception is working with the game’s code, since I don’t have a specialized education and work with code independently extremely poorly and slowly.
Looking at what modern vibe coders or AI sloppers are producing, I’ve long had the feeling that I’m some outdated old man who understands nothing about modern technology and has hopelessly fallen behind the more advanced youth. And I don’t really want to catch up with them, because I enjoy the process of creating a game by hand and don’t want to replace it with something more advanced and faster, but soulless.
That said, I really liked the acceleration in coding work provided by modern agents. In my day, I was absolutely thrilled by how much it sped up the overall work. And I’m still enjoying it to this day.
What do I use? Since I don’t need some fancy agent with an insane number of tokens, I use the free Big Pickle model (god knows what’s hidden underneath it, but it works well) or Deep Seek V4.1 through the open-source AI agent OpenCode. This work looks like some kind of archaicism from the 1990s. OpenCode does have a more familiar modern interface, but, as a proper old man, I’m simply too lazy to figure it out, since this ancient thing handles coding tasks perfectly well.
Each task is a new session. I suppose configuring OpenCode so that it remembers previous sessions isn’t that difficult. But I don’t need that either, since the project itself is already in my head, and I understand perfectly well where to direct the agent to implement any particular task. And for coding work, as a rule, Big Pickle’s daily free limits are quite sufficient.As a last resort, I have a second PC on which I can continue working on the project. And as an absolute last resort, a local Qwen 3.8 27B is installed on the PC. Although it works very slowly, it also works reasonably well through the same OpenCode.
But I’ll say right away that implementing a self-learning model is far, far beyond my knowledge, and I would not have been able to do it on my own. Just take a look at my current reward formula for the model so you can understand what I have to work with:
R_total = R_terminal + R_intermediate
R_terminal = +100 (victory) / −100 (defeat)
R_intermediate = clip( R_combat + R_damage + R_tactical + R_board , −80 , +80 ).
Where: R_combat | clip( λ_c · (Σ_enemy V(u) − β_c · Σ_own V(u)), −C_c, +C_c ) | λ_c = 10, β_c = 1.2, C_c = 70 | ±70 |
| R_damage | clip( λ_d · (Σ_enemy f_d·V(u) − β_d · Σ_own f_d·V(u)), −C_d, +C_d ) | λ_d = 8, β_d = 1.2, C_d = 15 | ±15 |
| R_tactical | sum of bonuses/penalties (see §5) | — | ±10 |
| R_board | clip( b_board · Δcells, −C_b, +C_b ) | b_board = 5, C_b = 10 | ±10 |
Classy, right? You see, everything immediately becomes clear and understandable when you look at a formula like this.
These are the starting conditions. And, unfortunately, even more than two years of work on the project have proved completely insufficient for me to understand how to implement this self-learning model. I had to involve an AI agent and a neural network much more deeply in working on this task.
3. Implementation of the self-learning model.
To begin with, my agent was tasked with conducting a full analysis of the implementation of combat by the opposing bot in the project.After the analysis, the same agent drew up a plan for introducing a self-learning model into the project, consisting of 20 sequential stages. This plan was provided to ChatGPT and, with its help, was refined and approved. I then proceeded with the actual implementation.
The work proceeded as follows: I fed ChatGPT the overall plan, reports on the completed stages, and the current state of the project, then asked it to create the necessary prompt for the current stage for my agent, which I passed on to the agent. The agent then carried out the assigned task and provided a report upon completion. Then everything started over. Every stage followed this same pattern.
I won’t say it was easy or that everything went smoothly, but overall, the work continued, the errors identified were fixed, and the work gradually kept moving forward.
First, all combat mechanics were completely copied and moved into separate scripts: attacking, movement, damage handling, card play, etc. In essence, another simplified game was created within the game, but without any visualization.
The next step was to create a script capable of running a combat simulation using all the copied mechanics. This script made it possible to run combat simulations at the code level, without any visual display, allowing simulated battles to execute very quickly (each one taking only a few seconds), while still being full-fledged battles.
Only after that was a trainable model added to the project. At this stage, the model was already able to interact correctly with the game logic, follow the rules, and choose permissible actions for attacking or moving.However, there had not yet been any proper training: the model knew the ultimate goal—winning in battle—but did not yet know how to effectively devise a strategy for achieving it.
The next important step was to implement the model’s learning mechanism directly. The battle simulation itself had already been implemented; now the model needed not merely to run battles, but actually to learn from them.
After completing this task and confirming that the model was indeed learning from the results of the simulated battles, I was delighted and concluded that the hardest part was behind us.
How naive I was. Everything was only just beginning…
4. The Main Bottleneck
First, I’ll note an important point. The self-learning models in my project are not large language AI models (such as Qwen, Chat GPT, or Claude) integrated locally into the game. Those giants have entirely different functionality that is simply excessive for my tasks. They are generalists, but their fucking capabilities are not needed in my game.
For my project, I use classical machine learning (Machine Learning). To explain the difference in two words: unlike general-purpose neural networks, my models are tailored exclusively to a specific task. For them, no world exists beyond my game. At their core are mathematical formulas and algorithms that are given access to game functions. During training, the program finds optimal patterns and builds its own base—that is the finished model.
But within my game, they will perform better than general-purpose models (unless, of course, those are also specifically trained on my rules).At the same time, the basic learning principles are roughly the same for classical ML and modern large AI systems—the only difference lies in the architecture and in the amount of information used for training.
Here, then, two parallel base models, A and B, were created for training, with each one trained separately. The model training process itself proceeds as follows:
- the base models A and B are frozen as fixed models.
- candidate A (a copy of model A with a self-learning configuration) plays many battles against the fixed models A and B;
- candidate B (a copy of model B with a self-learning configuration) plays many battles against the fixed models A and B;
- finally, the 2 trained candidates A and B battle each other;
- if progress is recorded, these trained models replace the frozen models A and B, and the cycle begins anew.
The key problem became apparent when the first serious run of simulated battles—several thousand matches—showed that the models really were learning. But they were learning to lose, because the win rate of the trained models in battles against the base models had dropped sharply (50% wins at the beginning of training and 30% wins at the end of training).
In other words, the experiment clearly demonstrated that the models really were learning from the battle results. But what they were learning wasn’t quite what I had expected of them.
I began investigating. I discovered that there was an incorrect sign at the end of the formula that reversed the training objective. In other words, the model was actually learning not to win, but to lose. I fixed it.The win rate of trained candidates against baseline models returned to 50%.
But after that, everything stalled. Thousands upon thousands of battles and numerous adjustments to my original formula led nowhere—the battle analysis showed that training itself was underway (the models’ control weights were changing), but this produced no visible result—the win rate did not increase and remained at around 50%.
I spent about 10 days implementing the battle simulation and preparing the models for training, and roughly the same amount of time trying in various ways to teach the models to produce a real result that would be recorded as wins, but with zero results. Whatever I did (and I am persistent), the outcome remained the same—simulations run over thousands of battles produced no progress: the win rate consistently did not change.
I think those who have dealt with self-learning models have already realized where I got stuck. I did not even suspect it....
And then the leather bags proved once again that it is too early to count us out. There is still powder in the powder magazines, and we may still prove useful in the emerging new world!
5. Finding a way out of the impasse
The aforementioned impasse was entirely caused by my stupidity. Even the much-praised neural networks could not tell me where my main mistake was.Moreover, I even turned to various neural networks (I gave Claude the same task, hoping it would figure it out and identify the reason for the lack of progress in training), but it was all pointless!
And these “smartest” neural networks, which know about game development and working with trainable models not just thousands, but tens, or even hundreds of thousands of times more than I do, still failed to identify my key mistake.
And the mistake was very simple: I made the win rate the sole criterion for successful training.
At first, it seemed to me that this was the right criterion. After all, how else could you evaluate the result of training than in the most tangible form—as an increase in the win rate of trained models compared with the baseline models?
And it turned out to be fundamentally wrong. Why?
I got the hint from somewhere entirely unexpected. During a conversation with a friend who has nothing to do with gamedev but enjoys watching various videos about current trends in scientific progress (including videos about working with neural networks), I complained to him about the deadlock in training. And at the very end of our conversation, my friend recalled that he had recently watched a video in which an indie developer was embedding a self-learning model into the racing game he was creating. That developer initially set winning the race as the criterion for successful model training. Nothing worked for him either, because he followed the same path I did.
Everything only got unstuck when he began breaking the model's training down into separate stages. In other words, the model was rewarded not for completing the entire race, but for successfully getting through each individual corner or obstacle.Whereas in the original training setup, where the model was rewarded only for fully completing the run, the cars did not even reach the finish line.
That’s it! It clicked in my head immediately. This was my main mistake!
The problem was that the model received its main and only reward solely for the final outcome—victory or defeat—which was achieved by destroying the enemy headquarters. Meanwhile, between the start of the battle and the destruction of the enemy headquarters, there was an enormous number of intermediate actions: deploying cards from the hand, choosing where to move and whom to attack, playing spell cards, using card abilities, accumulating resources to play cards, etc. And in my initial implementation, none of these received any additional evaluation.
Why on earth did I assume that my model would engage in all these physical movements if it had an overall goal—victory through destroying the headquarters? Well, the model pursued that goal in the simplest possible way—it destroyed the enemy headquarters with direct attacks, completely ignoring any tactics or other mechanics: it simply always attacked the enemy headquarters with its own headquarters, and all deployed cards blindly headed toward the headquarters and tried to attack it.
Because my training was conducted on training headquarters whose damage and durability stats were identical for all nations, victory went to whichever headquarters took the first turn and dealt the first damage to the enemy headquarters.And so, in the simulation, I implemented a mechanic in which the side that moved first alternated. As a result, the models would win alternately, depending on whose turn started first.
This explains the lack of progress and the overall battle rate of 50% for all training variants at the end of training.
I still don’t understand why on earth I decided that the model would be sensible enough to use more complex tactics to win than simply destroying the headquarters directly. As if I didn’t understand that AI brains aren’t real brains, but merely an imitation of them and a treasure trove of knowledge. I understood that, but still wasted a lot of time on absolutely useless work.
On the other hand, this sack of skin, despite knowing nothing about the subject, immediately found the potential cause of the error just by recalling an episode from a video he had once watched!
6. What comes next?
In my game, there are various ways to destroy the enemy headquarters and thereby win a battle. For example, by assembling a deck specifically designed to destroy the enemy headquarters. But the most popular method—and my preferred one—is battlefield control, achieved by effectively destroying the enemy’s deployed tank cards.
According to my new idea, the model should learn to win precisely through effective battlefield control. Therefore, I adjusted the model’s reward formula by adding rewards for the number of enemy vehicle cards destroyed and a penalty for the loss of its own cards.Now my reward formula looked as follows: R = episode_reward + 0.6 clip(kills - 1.2 losses, -3, +3), which allowed the models to be rewarded for a positive kill balance.
At the same time, so that the model would not forget the main objective of combat—winning the battle—the additional reward for kills was set high, but lower than the reward for destroying the enemy headquarters. Otherwise, there was a risk that the model would endlessly destroy the units deployed by the opponent to score points, without striving to achieve victory.
And this produced results. Now, during training, the models consistently began to outperform the baseline models in terms of the kill-to-loss ratio. This, finally, allowed me to replace the baseline models with trained ones, thereby making progress in training.
However, despite the result obtained, it was clear to me that these changes were insufficient. And using a positive balance of destroyed and lost units as the main criterion for training effectiveness did not give me confidence that the final result of training the models would satisfy me.
The result of the new brainstorming was another adjustment to the reward formula, which now looks as I showed earlier.The new reward system already accounts for different gameplay situations:
- Destruction without retaliation (the target is destroyed, while the attacker [unit] does not lose HP);
- Ranged destruction by an SPG (when the attacker is an SPG and the target is destroyed);
- Ranged destruction by an HQ (the same thing, but for an HQ);
- A one-strike kill of a tank destroyer (this makes it possible to reward the use of the tank destroyer game mechanic, which takes no counter-damage if the target is destroyed during the attack);
- Killing a heavy tank during its cooldown (another reward for using the heavy tank game mechanics);
- A failed attack (as a result of the action, your own unit dies, while no enemy is destroyed or damaged);
- Damage dealt to an enemy HQ (a small constant bonus);
- Control of the enemy bridgehead (to make it harder for the player to play cards from hand).
The results of the runs under the new reward system seemed quite encouraging to me. All the additional metrics are steadily improving across the runs, and the models being trained outperform the baseline models on these metrics.
Of course, it is still too early to draw unconditional conclusions about the future success of the entire effort to create truly “smart” models. But now there is much more confidence in a positive outcome.
It would seem that I have finally worked out the main learning challenges, and now I can calmly move forward. But who would stop me from making the same mistakes again? Right—nobody!
7.Stumbling Block No. 2
How can we avoid yet another stumbling block? And what do you think caused it? Exactly—I started thinking too late again! I trusted the neural network once more, and once again, that was a mistake. Don’t be like me—think about what you’re doing! Hopefully, this will be another clear example of how not to do things.
As I’ve already said, at first I mistakenly chose the increase in the win percentage of trained models compared with baseline models as the criterion for successful model training. Later, after the reward system was changed, the models began training in a more targeted way, and their progress became apparent.
However, I still relied on the win percentage as the final confirmation of the effectiveness of training. But the win percentage of the trained models in fights against the baseline models, as before, fluctuated within the margin of statistical error.
And every time after another training session, the neural network would jab at me with the final report, insisting that there was still no proven effectiveness in model training because the win percentage had barely changed.And the neural network and I spent a lot of time discussing what could be adjusted in the training settings to increase the win rate of the trained models.
Collectively, we “discussed” it! “Colleagues,” damn it! Well, the neural network has no brain; what do you expect from it? But how could I have been so dumb?
The main mistake in the analysis was that each time, the effectiveness of the trained models was compared in battles against the baseline models, and the baseline models were updated every day after every successful training session!
Thus, in effect, each new version of a model was not being compared with its original level of capabilities, but with an essentially identical model that was simply one training cycle behind.
And naturally, the difference between the trained model and exactly the same model that had undergone just one fewer training cycle was negligible! It would have been enough to use your brain for even a moment and think: where the hell would any significant difference come from if these were practically the same models? After all, after every new training session, I replaced the baseline models with the trained ones!
Naturally, once I realized this completely uncomplicated fact, I immediately created separate control models that would not be updated and would serve as test subjects. These control models were built into my learning system, and the effectiveness of the already-trained models will be tested against them.
8. Conclusion
At present, the model training process is in full swing (and this is no quick process).At the same time, I am working on embedding the learning mechanism itself directly into the project so that the training process can be launched straight from the project’s engine. I also periodically adjust the reward system so that the model learns to use different game mechanics properly, rather than merely flailing around during combat. And I very much want to believe that this is bringing me ever closer to the main goal: obtaining a sufficiently “smart” model that can give a real person a real fight.
I think many people will ask: why the hell did you even write this article if your own work on training the model is not yet complete?
In earlier articles, I already expressed the idea that the development of every individual game project is too specific for the implementation of particular game mechanics to be transferred to other projects, even those in similar genres. Therefore, it makes no sense to provide detailed instructions on integrating a self-learning agent into your game: they are unlikely to be useful for your particular project.
However, the mistakes I made are universal and very illustrative. They can occur in the work of anyone who relies too heavily on the “brains” of neural networks.
And it would be great if my article helps someone who is already creating a learning model for their game or is considering whether to add a self-learning model to their project.And perhaps someone will get the idea to add a self-learning model to their game as well.
Let this assistance take the form not of step-by-step instructions, but of conveying the main idea:
- Even (especially) when using neural networks, think about what you are doing!
I really hope that my article helps you avoid wasting time on useless work, as happened to me. Remember that it is your brain that must be the main tool in the work, and that the final result of such work depends precisely on the use of your gray matter.
Thank you to everyone who read the article to the end.
I would also be grateful for any constructive advice and indications of mistakes I made in my work.
Good luck to everyone.
P.S. A link to the demo of my game on Steam and to the game's Telegram chat.
P.P.S.And, as always, a few screenshots from my game
