We gave Claude the documentation, and it started writing automated tests for us / Habr

*RTFM: Read The F*** Manual

Hello everyone! I’m from the Документерра development team. We build a platform for documentation websites, so I could talk about documentation for hours. But today’s story isn’t about the product. It’s about how documentation that testers used to review whenever they had time became the main source for their automated tests.

How it used to be

For as long as I’ve been writing software, documentation has lived at the very end of the pipeline. We write the code, ship the feature, and someone documents it. Customers read the documentation. Inside the company, it was needed by practically no one: developers already know how everything works (or at least, that’s what they think), and testers were asked to review it as a favor whenever they had time. And as you can imagine, that time was rare.

How it is now

This year, documentation gained a reader inside the company. Testers started feeding it to Claude and getting test cases and automated tests in return. Documentation moved from the end of the pipeline to the beginning: now testing starts with it.

Here’s what we can do now. At the beginning of 2025, I would have considered half of this list overly optimistic.

• Automated tests without wrestling with selectors. One hour of prompting, five hours of Claude’s work, and we have working automated tests for a feature. No one has to find the selectors or manually come up with the steps. All we feed in is the test case description.

• Tests for an unfinished feature. We write and fix tests by giving the documentation to the model.If a new feature relies on existing functionality, the test that sets up the preconditions can be written even before the new part is ready.

• “Test it and write automated tests.” We give the model a new feature, a PR, a Jira task, and documentation. There is no longer any need to manually click through the feature first to understand how it works and what needs to be tested.

• Tests for a feature the tester has never laid eyes on. If there is documentation for it, Claude will figure it out.

• Hunting down outdated test cases. We find manual and automated test cases that contradict the current documentation and update them.

• Bugs in bulk. The model usually realizes it has encountered a bug from the test cases, the documentation, or simple logic (an error has appeared, the button does not work). Another source of bugs is analyzing the logs from test runs.

Here is how this looks in practice, using a simple feature: password reset.

Step 1.Documentation. The model received the following fragment via the MCP server:

1. Open the portal and click the Log in button to proceed to the login page.

2. On the login page, click Forgot password?.

3. On the password recovery page, enter your username or the email address associated with your account and click Reset password.

4. You will receive an email with a link to set a new password.

Step 2. Test cases. They were prepared by an AI agent. The precondition for both: the user is not authenticated.

TC-1. The “Forgot password?” link opens the password recovery page (steps 1–2 from the documentation)

Step	Expected result
1. Open /login/	The login form has a Forgot password? link.
2. Click Forgot password?	The /forgot-password/ page is open with a username field and a Reset password button.

TC-2.Without a login, the reset request is not submitted (step 3, negative)

Step	Expected result
1. Open /forgot-password/	The login field is empty
2. Click Reset password without entering a login	A validation error appeared in the field, and the page did not change

Please note TC-2: the documentation says nothing about an empty login. The model derived the negative test case itself, from the form's standard logic.

Step 3. Automated test.

import {test, expect} from '@/fixtures/test'; import {ForgotPasswordPage} from '@/pages/Application/forgotPassword.page'; test.use({storageState: {cookies: [], origins: []}}); // anonymous user test('TC-1: "Forgot password?" opens the recovery page', async ({page}) => { const forgot = new ForgotPasswordPage(page); await test.step('1. Open the login page', async () => { await page.goto('/login/'); await expect(forgot.linkOnLoginPage).toBeVisible(); }); await test.step('2. Click "Forgot password?"', async () => { await forgot.linkOnLoginPage.click(); await expect(page).toHaveURL(/\/forgot-password\/$/); await expect(forgot.loginInput).toBeVisible(); await expect(forgot.resetButton).toBeVisible(); }); }); test('TC-2: an empty login is not submitted', async ({page}) => { const forgot = new ForgotPasswordPage(page); await test.step('1. Open the recovery page', async () => { await forgot.goto(); await expect(forgot.loginInput).toHaveValue(''); }); await test.step('2. Click "Reset password" without a login', async () => { await forgot.resetButton.click(); await expect(forgot.loginError).toBeVisible(); await expect(page).toHaveURL(/\/forgot-password\/$/); }); });

The pipeline, or where we don't trust the neural network

All of this came together into the following pipeline:

TAG205@@ Two mandatory reviews and oversight at the automated testing stage: the model writes test cases and code, but the decisions remain with a human.

There are two review gates here. Both are manual, and neither can be skipped. Let me explain why.

TAG210@@Documentation review. Everything downstream in the pipeline inherits its content. If the documentation describes behavior that the product no longer has, the test cases will codify it, and the automated tests will start checking it. You get the classic scenario: two systems agree with each other, and both are wrong. The tests are green, everyone is happy, and meanwhile, the product quietly drifts off course.

Test case review. This gate is cheaper and catches something different: gaps in coverage. A person who knows what customers do with the product sees them right away. A model that works with text does not.

TAG218@@The third stage requires less oversight than I expected. And an unexpected conclusion: it's more useful to read the chain of reasoning than the generated code. It shows that the model was thinking about the product before it wrote the first line.If it has misunderstood any part of a feature, that is immediately obvious, without digging through the code.

“Why not just export it?”

Testers read the documentation through the MCP server for DokuTerra’s published documentation. The first question we are asked is: why use MCP if the documentation can simply be exported? There are four reasons.

• Volume and coverage. For large volumes of private documentation, MCP is essentially the only workable method. You can export it, but the process is slow and tedious, and beyond a certain size, it is not possible at all. Most importantly, when a test case involves several related parts of the product, no one knows in advance which pages will be needed. You either have to export everything or guess. Through MCP, the model finds what it needs on its own.

• Private documentation. Much of what testers need has not yet been made available to users. A feature is tested before its release, so its documentation is still internal. Export workflows are generally designed for content that is already public. Yet the MCP server serves such pages as well: we keep pre-release feature documentation in private publications, and only those with access can read it through the server.

• Currency. The most important reason. When the model reads through MCP, I know it is working with our current documentation, rather than with what it memorized during training or found in a random article about our product.And this is verifiable: in the chain of reasoning, you can see when the model queried the MCP server.

• People don’t know how to explain. The fourth reason has nothing to do with tools. Ask an engineer to write the context for the model, and you’ll get something incomplete. Not because the engineer is bad, but because he knows too much and doesn’t notice what he left out. The documentation was written for someone who doesn’t know. Therefore, as a source of context, it is better than a person who does know.

The server connection looks like this:

Work on a new feature starts with the test-design skill that we created for an AI agent. You give it everything known about the feature: the developer’s How to test, a link to the task, a pull request, existing test cases, or simply a description of the task in your own words. It reads the product documentation itself via MCP. The result is a ready-made test plan with test cases, which is then used to write the automation. Here’s how it is invoked:

What Documentation Does the Neural Network Like?

The experience of the testers made me reconsider what good documentation means.

For testing, documentation does two things. It describes how the product should behave, and test cases are derived from this. It also describes how to perform common actions in the product. This helps the model navigate the interface and unpack high-level cases such as “create a project and publish it.”

Now let’s talk about what exactly makes documentation usable by a machine.

Constraints, not descriptions.

TAG280TAG282

TAG284@@Useless for tests

TAG287TAG295TAG300

Files larger than 50 MB are rejected with error 413

TAG304TAG306TAG308

Every precisely stated boundary becomes a check that nobody has to invent.

Here is how it looks in practice: the wording seems precise, but the model finds three questions in it that must be answered before a test can be written.

TAG314@@Yes, this is our own documentation. We build a documentation platform, and we write things like “every 30 days” too, sincerely believing that this is precise wording.

Scenarios, not buttons. A page about where each button is becomes outdated at the very first change to the interface, and every test derived from it breaks along with it. A page about what the user wants to accomplish survives a redesign. The test survives too: the model derives the steps anew for the new interface, because it knows where it needs to get to.

For years, our technical writers fought for documentation built around user tasks. Their argument was this: a client comes with a task, not out of curiosity about the menu. I nodded, agreed, and considered it a matter of taste.It turned out that this determines whether the test suite will survive the redesign. Colleagues, you were right.

Release notes. They tell us what has changed, and therefore what needs to be regenerated or fixed. Without them, we would have to compare the behavior and figure out what has shifted.

We don’t need to write more. We need to write more precisely where vagueness used to get away. 

Who takes care of the documentation now?

For years, the relationship between documentation and testing worked in one direction: testers checked the documentation out of politeness. Now everything is the other way around. Documentation has become an input for their own work, and its quality directly determines how much automation they will have to rewrite each quarter. Testers now have their own, rather self-interested reason to care about it.

The same thing is beginning to happen with development, although we’re still at the beginning of that journey. We plan for Claude to consult the public documentation during development: when uncertain, so as not to break backward compatibility and to understand what users actually do with a feature. In essence, the documentation describes the product’s expected behavior. It is still far from a true specification: it is high-level and not very detailed. But for most features, there simply is no other description, and that is better than nothing.

I don’t think this applies only to us. If documentation powers test generation, support responses, and internal tools, the debate over whether it is worth maintaining is no longer a debate about craftsmanship.It becomes an expense item whose disappearance several teams would immediately notice.

What hasn’t taken off yet

We haven’t been using the MCP server for long, so everything written above is an early result, not established practice.

Images. Our documentation contains screenshots, and some of the information in them isn’t duplicated in the text. The MCP server returns Markdown with links to images, but we haven’t checked whether the neural network does anything useful with them. If a substantial part of the knowledge about your product lives in screenshots, for now assume that such a pipeline doesn’t have them.

Cyclicality. I’ll repeat this because it’s the most expensive risk. Documentation generates test cases, and test cases cement behavior. If nothing in this loop is tied to the product itself, both sides can drift away together, in complete agreement with each other. That’s why we need a manual review gate, and I wouldn’t remove it to save time.

Instead of a conclusion

The main thing I took away from this year is that documentation has stopped being a report on completed work and has become a tool for what comes next. And for the first time, it has internal readers who need it not out of courtesy.

Are you already feeding your documentation to AI systems? It’s particularly interesting if someone has taught a model to work properly with screenshots from the docs.Tell us in the comments, we really need it.

TAG289Ready-made test caseTAG293

TAG297@@The system processes a file