LLMs are increasingly being integrated into existing business processes: information search and collection, data structuring, verification, and publishing. At the same time, the cost of a single mistake can be high: lost money, time, and reputation.
The Human-in-the-Loop principle helps reduce this risk: a person who validates the data is added to the process. This article brings together our findings from two real-world business tasks: automatic translation and size chart search.
Context: business tasks
Our findings are based on two tasks in which we evaluated LLM results:
Product translation. Product names and descriptions translated from English into Russian, tens of thousands every day. Several different models + an LLM judge + script-based validation are used. See the previous article for details.
Size chart search. We search for a chart using the combination “brand + product type + size + gender” (for example, “Nike, sneakers, 11 US, men’s”). Two models with web search independently find the size chart on the brand’s website and return values in centimeters; the judge compares the answers, and a human performs the final check.
Conclusion 1: Start with Deep Research
Before starting any work, Google it and run a deep search, preferably using different agents. Deep Research is implemented differently in Claude, Gemini, and ChatGPT, and the results of the three research efforts complement one another well. This way, you will see most of the problems you will encounter and the solutions already available in advance.
Ask it to format the results as an MD file and place it in the project folder.The agents you will work with next will be able to read it and take information from it.
Research prompt: formulate it together with the LLM: describe the task and ask it to help formulate exactly what needs to be clarified.
In our tasks, we found a wealth of useful information on automated translations (a very popular task) and little useful information on searching for size charts (a rare task)
Conclusion 2. Assess the cost of an error and the volume of the task
Assess the task in two dimensions: what one error costs and how many records there are per day. The verification strategy depends on this.
Small volume | Large volume | |
|---|---|---|
Costly error | A person checks every record. A convenient verification tool is needed.(size charts) | Automated checks filter out the obvious cases, while a human handles the questionable ones. |
Inexpensive error | You can spot-check or not check at all. | A judge model, deterministic checks, spot audits, and monitoring.(translations) |
A translation error is inexpensive: the English original and photos are available alongside it. There are tens of thousands of products per day, so they cannot be checked manually.
An error in the size chart is costly. If the insole is listed as 260 mm instead of 250 mm, the sneakers will not fit, and there will be a return.The volume is moderate, and a person can check every record.
Conclusion 3. Involve experts when choosing models and prompts
Involve experts at an early stage. Together with category managers, we evaluated the quality of translations and the sizes found. Run 50–100 records through different models with different prompts and present the results to experts in a convenient format so they can compare the options and choose the best combination. We ran records through three models with several variations of prompts and sent the results as an HTML page where names, descriptions, or the size grids found could be compared on one screen. The experts identified inaccuracies in the translations of cosmetics and medications, which we corrected using prompts.
Test specific “model + prompt” combinations: the same prompt works differently across different models. Each provider has its own prompting guidelines (Anthropic, OpenAI, Google), and these even change within a single product line: the rules for writing prompts for Opus 5 differ from those for Opus 4.
Separate prompts by category. For translations, we use different prompts for different product categories, while size grids have separate rules for footwear and clothing. Before sending the request, the script assembles the final prompt from the common base and the relevant branch.
Conclusion 4. Give the model maximum context
At first, we translated only the product name. It is often ambiguous: a string of words without prepositions, abbreviations, or product line names. We added the description, and the quality improved. In the end, we settled on providing the full context: website, product link, category, size, name, and description.
During Deep Research, we found similar cases among major companies.At Walmart, the one-letter attribute value “N” (an abbreviation for “No”) was translated as “North” until the attribute name was added to the prompt. At DoorDash, many LLM-judge errors actually turned out to be a lack of information: prices, menu settings, and display logic. If the model or judge makes a mistake, first check whether they have enough information.
Takeaway 5. Require the model to provide explanations and evidence
Add a field to the model’s response containing the rationale for its decision. In translations, this is an explanation of which rules from the prompt the model applied (we have different rules for different categories). In size charts, this is evidence: a reference to the HTML block of the page containing the chart, a link, and the verbatim text excerpt from which the value was taken.
There are fewer errors: the model hallucinates less and tries to justify its answers by relying on a specific rule or excerpt. Place the rationale field before the final value so that the model reasons first and then answers. When analyzing errors, it is immediately clear where things went wrong, and evidence from the source speeds up manual review.
Takeaway 6. Use an LLM judge that responds only with “yes” or “no”
Translations have two stages: first, the model translates; then, the LLM judge checks the translation and improves it if necessary.
In size charts, two models, OpenAI and Gemini, independently find the chart. A third model, Claude, acts as the judge: it compares their results and decides whether the chart can be accepted. This gives us two signals: whether the two models’ responses match and whether the judge approved them. If at least one does not match, we examine the record more closely.One match is not enough: different models often make the same mistake.
We initially used a five-point scale, but found no difference between scores of 3 and 5, so we left it as “yes / no.”
It is also pointless to ask a model to assess its confidence in its own work. A model with a score of 10 out of 10 gave us a value from the women’s size chart instead of the men’s.
Conclusion 7. Best check: deterministic scripts
In American listings, dimensions and weight are given in inches, pounds, and ounces, often directly in the title. When translating, these need to be converted to the metric system. The model periodically makes mistakes: it leaves inches as they are, converts incorrectly, or omits a value.
We catch such errors with regular expressions: we compare the numbers in the original title with those in the translation. If the translation contains the same number as the original, the unit has not been converted or has been converted incorrectly, and the translation is sent back for revision.
If values can be converted by a script without an LLM, do that: extract them with regular expressions and recalculate them using conversion factors. Give the model only what the script could not convert, or add new cases to the regular expressions.
Conclusion 8. Record the model version and prompt version
Store the model version and prompt version with every result. If you find a defect, you can immediately see which results were produced by the same combination and what needs to be rechecked.
When switching to a new model or prompt, run a validated test set and compare the results of the old and new combinations manually or with the LLM judge from Conclusion 6. The new model may perform worse on your task.
Conclusion 9.For manual verification, create convenient tools
When working with size grids, managers use Chrome and brand websites, so it makes sense to check grids in Chrome as well. To do this, we created an extension that:
takes JSON containing LLM results as input;
- TAG156@@opens the source page, scrolls to the relevant location, and highlights the value found OR, if the grid is not found, immediately opens a Google search with a specific query
- TAG160@@allows the reviewer to confirm, correct, or reject the result using hotkeys; TAG163
outputs JSON that is automatically uploaded to the admin panel (after authorization).
TAG167Colleagues work in a single tool and do not need to switch between multiple windows. Each action takes one click. The verification speed has increased dozens of times.
Conclusion 10. Become the reviewer yourself
To make a truly convenient tool, you need to understand the context firsthand, and the best way to do that is to work with it yourself for a while. After each new version, sit down and spend a day or two in the role of the person for whom you built the tool. This will immediately reveal all the rough edges and possible ways to improve it.
Conclusion 11. Regularly check manually and collect feedback
Models are updated and data changes, so without regular verification, quality gradually declines. Once a month, check the results manually and collect feedback: user complaints and support requests.
Conclusion 12.Use errors and complaints as a signal to improve the system
We analyze reported errors and complaints and amend the prompts. If an error occurs repeatedly, we rework the prompt by adding exceptions and examples.
At Walmart, linguists are prohibited from making manual corrections to translations: they formulate a rule, a second linguist validates it, and the rule is applied to the entire catalog.
The more context you accumulate about your system, the better it performs: error → analysis → prompt correction → fewer such errors.
Conclusion
At present, LLMs can be entrusted with processes where the cost of an error is low or even moderate. However, critical processes should be checked manually until you have accumulated a large body of positive results—for example, until the system has run for several months without errors.
By gradually improving the system, you will most likely reach this outcome, but debugging will take a considerable amount of time. The tips above will help you complete this journey faster and more efficiently.Thank you for your attention!
