Model annotates, code counts: how the AI checking for the EGE in social studies is structured / Habr

The model annotates, the code counts.

I am a teacher and an expert in the EGE, for about two years I have been working on a Telegram bot that checks answers to the second part of the EGE in social studies, and the main thing this project taught me: models cannot be entrusted with scoring.

In its current form, the bot works as follows: the model does what it does well — reads the answer and annotates it according to the scheme: which examples are considered specific, which statements are correct, which points are key. Then the regular code takes over. It recalculates the scores according to the criteria, checks what can be checked without the model, and initiates rechecks, which can only lower the score. In this part, I will explain how it works: the path of one answer, the division of responsibilities between the model and the code for each task, two code fragments, reliability and cost. How accurate this whole process is and how I measured it, I will tell in the second part.

What exactly is checked

The EGE in social studies consists of two parts. The first part — tasks 1–16 with short answers — is checked by a machine. The second part — tasks 17–25 with extended answers — the student writes a text, and it is evaluated by two independent experts according to the FIPPI criteria. If their scores differ significantly, the work is reviewed by a third person.

Task

The complexity for AI is that the criteria are very formal. The example in task 19 must be specific: "the police maintain order" is a function, not an example, and it gets 0 points. If the task refers to "Russia," an example about "state Z" is not counted. The plan in task 24 that lacks three mandatory points from the reference receives 0 points entirely, even if the rest is correct. The model that "generally understands social studies" evaluates such answers too generously. Hence, the architecture described below.

Path of one answer

The primary score is set by the model, and everything after that can only reduce it. Inside the bot, all steps are regular await in a single process, there is no queue. The same evaluator is called from the teacher mode, from the website, and from the B2B API; in B2B, there is a Redis queue before it.

Who calculates what in tasks 17–25

Each task has its own plugin: handler, evaluator, and specification. There is no common "universal prompt for the second part": the criteria for the tasks are so different that the share of work done by the model varies, from "solves almost everything" to "does not affect the score".

In all tasks except one, the code can only lower the model's score. The exception is task 20: here the code recalculates the penalty for extra judgments and can increase the score if the model left the answer blank where one point is deducted according to the rules. The model does this consistently.

What is the "markup"

The model does not write in the answer "I give 2 points because...". Instead, it fills in the JSON schema through tool_use, and then the code checks each field in it. Here is a shortened schema for task 19:TASK19_PRIMARY_SCHEMA = { "type": "object", "additionalProperties": False, "properties": { "score": {"type": "integer", "minimum": 0, "maximum": 3}, "total_examples": {"type": "integer", "minimum": 0}, "task_requirements": { "type": "object", "properties": { "requires_russia": {"type": "boolean"}, "requires_structure": {"type": "boolean"}, }, }, "valid_examples": { "type": "array", "items": { "type": "object", "properties": { "number": {"type": "integer"}, "text_snippet": {"type": "string"}, # quote from the answer "is_about_russia": {"type": ["boolean", "null"]}, "why_valid": {"type": "string"}, "illustrates_thesis": {"type": "boolean"}, }, }, "invalid_examples": {"type": "array"}, }, }

The most useful field here is illustrates_thesis. The model often counts examples that are specific and factually correct but illustrate the wrong topic: for example, talking about elections when asked for an example of a source of information about the legal system. A separate flag forces it to answer this question explicitly, and the code then limits the score by the number of examples where the flag is set to true.

Task 19: recheck that only lowers the score

In Task 19, the student provides three specific examples. Some formulations require showing action of a state organ, and here the model systematically counted irrelevant examples: retelling the function of the organ or a story about what a citizen did. The solution is a second model call with a separate schema, which the code resolves on top of.

First solution: when not to run the recheck. The first version triggered on all tasks and deducted points where family or citizen actions were involved and there was no organ in the condition. Now it is triggered only by explicit markers in the task text:

def _task_requires_organ_actions(self, task_text: str) -> bool: """Conservative default — return False in the absence of explicit positive markers to avoid false rechecks (−2 points penalty). """ text = (task_text or "").lower() organ_patterns = ( r"принуждать\s+к\s+соблюдению", r"правоохранительн\w*", r"институт\w*\s+принуждения", r"контрольно-надзорн\w*", "illustrates_thesis": {"type": "boolean"}, }, }, "invalid_examples": {"type": "array"}, }, }

Task 19: recheck that only lowers the score

def _task_requires_organ_actions(self, task_text: str) -> bool: """Conservative default — return False in the absence of explicit positive markers to avoid false rechecks (−2 points penalty). """ text = (task_text or "").lower() organ_patterns = ( r"принуждать\s+к\s+соблюдению", r"правоохранительн\w*", r"институт\w*\s+принуждения", r"контрольно-надзорн\w*", r"ensuring compliance", r"application of law", ) if any(re.search(p, text) for p in organ_patterns): return True return False # income redistribution, factors, sources of information — missed

Second solution: one signal is not a reason. The model assigns four flags to each example and rejects it only when there are two negative signals. The score can only decrease:

# High-confidence reject: an example is rejected ONLY when ≥2 independent # negative signals are present. A single flag is too noisy. strong_rejects = [] for e in examples_list: if not isinstance(e, dict) or e.get("passes_recheck") is True: continue negative_signals = sum([ e.get("organ_is_actor") is False, e.get("has_concrete_action_verb") is False, e.get("is_abstract_function") is True, e.get("is_citizen_narrative") is True, ]) if negative_signals >= 2: strong_rejects.append(e.get("number")) n_strong_passed = len(valid_examples) - len(strong_rejects) if 0 <= n_strong_passed < len(valid_examples): new_score = min(original_score, n_strong_passed) # only downward if new_score < original_score: primary_result["score"] = new_score primary_result["penalty_applied"] = True

Both solutions address one issue: protection against overestimation itself leads to underestimation, and the area of its application is as important as the check itself. I will discuss how this affected the numbers in the second part.

Assignment 24: the score is set by the code, the model only disputes

Assignment 24 requires creating a complex plan on the topic. The first criterion checks whether the plan contains mandatory points, and it's almost a mechanical check. Therefore, in bot K1 and K2, the code is completely counted, and model calls only affect the text analysis.

K1: lemmas and synonyms of headings

Plan points are lemmatized with pymorphy2 and compared with the bank's reference plans (185 topics). The threshold for matches depends on the length of the key point. The most annoying error in this scheme is synonyms: a student writes "Признаки" (Features), while the reference has "Особенности" (Characteristics), and the correct plan gets zero points. Therefore, the "heads" of the headings are brought to one form:

_HEAD_SYNONYM_GROUPS = ( frozenset({'признак', 'особенность', 'черта', 'характеристика', 'свойство'}), frozenset({'вид', 'тип', 'типология', 'классификация', 'разновидность'}), frozenset({'этап', 'стадия'}), ) _HEAD_CANON = {lemma: min(group) for group in _HEAD_SYNONYM_GROUPS for lemma in group}

Two precautions prevent synonyms from inflating the score. Synonym-based checks are only applied to points that did not match literally, and one student point cannot cover more than one key point. Moreover, the distinctive word, apart from the "head," should still be in the heading: a bare "Классификация" (Classification) cannot cover "Типы партийных систем" (Types of party systems). The quality check is simple: all 185 reference plans, when submitted as answers, should receive maximum scores according to K1. Currently, they all do, but before the correction, when headings were replaced with synonyms, only 147 of them did.

K2: four filters before deducting points

K2 is automatically assigned when K1 is fully met. It can only be deducted due to a factual error in the plan, and here the model was the most dangerous source of under-scoring: a single "find errors" query would deduct points even from reference plans. Now the finding passes four filters:

1. error type — from a closed list (currently there is only one type: non-existent type or function);

2. verbatim quote of at least 8 characters, which actually exists in the plan;

3. two independent runs pointed to the same fragment;

4. a separate run-verification confirmed that the mentioned type or function is not present in the school curriculum.

The fourth filter appeared last, and it is the most subtle. Agreement between two runs captures not only the student's error but also the common false model stereotype: on the reference plans, both runs jointly found the same non-existent error. Two identical answers from one model are not two independent opinions.

def _deductible_findings(findings, plan_text): """The quote must be exact — a paraphrased 'error' cannot be shown to the student nor compared between runs.""" plan = _normalise_quote(plan_text) kept = [] for finding in findings: if finding.get("error_type") not in K2_DEDUCTIBLE_ERROR_TYPES: continue quote = _normalise_quote(finding.get("quote")) if len(quote) < 8 or quote not in plan: continue kept.append(finding) return kept

Findings that did not pass the filters are not discarded: they end up in feedback but do not affect the score. Evaluation and feedback are separated.

When AI is unavailable: it is better to honestly refuse than to invent a score

The most unpleasant error in such a system is unnoticed. The model did not respond or returned an unreadable JSON, and the code silently assigned a zero, so the student sees a regular analysis with zero points. My task 25 worked this way: 16 such zeros out of 110 cases, and half of them matched the reference and were counted as "exact hits" in the metrics.

Now, in most tasks, any failure is reduced to one flag ai_unavailable:

• no free slot within 90 seconds (no more than 6 concurrent requests per process);

• network error after three attempts with exponential pause;

• model refusal (stop_reason: refusal);

• invalid JSON after repair and one retry with strict instruction.

With this flag, the score is not shown at all. The student sees "Check temporarily unavailable... Attempt not counted," and the administrator receives an alert. The order also matters: the free check is counted after delivering the result. Previously, it was the opposite, and any message sending failure in Telegram would burn the student's attempt.

In the November run on 393 cases, there were no silent zeros, model refusals, or parsing errors. But honestly, I have to say it's not fully implemented everywhere: in tasks 21 and 23, and on the path of task 24 for the website, a simplified score is still assigned without the flag in case of failure. There is no automatic switch to a backup provider: YandexGPT is manually turned on via environment variable and restart.

Structured response without forcing

A small detail that will be useful for those working with new versions of Claude. On the current model, forced tool_choice returns a 400 error, so the tool is passed with {"type": "auto"}, and a request to call the tool is appended to the end of the query. If the model still responds with text, the JSON is extracted from the text and fixed if necessary. Temperature is not passed on new models; instead, the effort level (effort: medium) is set.

Price and speed: about two cents and nine seconds per response

In November, a full run on 393 responses made 497 model calls and cost $8.87. On average, one check requires 1.26 calls, $0.023, and 9 seconds of model work.

* Task 24 here was measured on the site path, where the model marks the plan items. In the bot there are more calls, I didn't measure its price separately.

Two observations. The most expensive is not the longest evaluation, but the rechecks: task 23 has almost two calls per response. And the prompt cache significantly reduces the cost: in task 19, two-thirds of the input is taken from the cache, while in tasks 17, 21, and 22, it doesn't trigger at all—it seems their system prompts are too short for caching.

Outside the table remain OCR of photos and delivery to Telegram. I haven't measured the "from message to score" time in production yet.

Stack, scaling, and what I would have done differently

The bot is written in Python 3.10 and python-telegram-bot 22. The site and B2B API are FastAPI, the B2B queue is arq over Redis, data is SQLite in WAL mode and PostgreSQL for B2B. Lemmatization is pymorphy2, retries are tenacity, the model is Claude Sonnet 5.5. Everything runs on a single server in several systemd units behind nginx.

Regarding volume: the core is around 33 thousand lines, tasks 17–25 are another 33 thousand, tests are around 1.8 thousand without counting other subjects. I largely delegate the implementation to Claude Code. My part includes criteria, benchmark markup, edit specifications, and resolving disputed cases.

Why Claude and what happens with student data

This is the first question I get, so I'll answer right away. The bot initially used YandexGPT, and until July 2026, the grading was done on it. I switched for three reasons, all visible in the numbers. On the same 202 answers, Claude matched the expert in 65.8% of cases, while YandexGPT did so in 54.5%. YandexGPT sometimes refused to answer completely normal exam tasks ("I cannot discuss this topic"), and such refusals resulted in zero points. Also, for structured answers, Claude has a server-side JSON schema, while with YandexGPT I had to parse the answers using heuristics. A detailed comparison of this is in the second part. YandexGPT remains in reserve.

What goes into the model: the task text, the student's answer, and the reference answers from the task bank. There is no name, username, or Telegram ID in the query. In student mode, the text of the answer after checking is not saved: only the score and the length of the answer remain in the database. Because of this, I can't measure accuracy on real user answers, but I consider this compromise correct.

Why not fine-tuning

The second common question: why so much code around the model if we can fine-tune it. I thought about this and even wrote a pipeline template for YandexGPT: exporting verified answers, uploading to the cloud, running fine-tuning. But it never made it to production, and now I don't consider this an error.

There are three reasons. There is not enough data: to make fine-tuning effective, you need thousands of labeled answers for each task, but I only have hundreds. The criteria from FIPИ are refined almost every year, and a fine-tuned model would have to be retrained, while I can fix the rule in code in the evening. Most importantly, the model's errors in evaluation turned out not to be "lack of knowledge in social studies", but violations of formal rules: counting a function instead of an example, deducting points for a non-existent error. Such rules are easier and more reliable to record in code than to rely on the model learning them from examples. The example with task 20 above clearly shows this: the model consistently confused "minus one point" and "zero" until the code took over the arithmetic.

What would I do differently if I were starting over:

1. One pipeline for all tasks from day one. Some tasks go through a common checking engine, others through their own handler. In the end, the same logic — tracking attempts, handling errors — has to be maintained in several places, and it inevitably diverges.

2. Regression set before the first prompt. The bot was launched in the first version in the summer of 2025, and systematic measurements appeared only in the spring of 2026. The first run showed that the bot overestimates the score for approximately every third answer.

3. Store the history of checks. Now the bot saves only the score and the answer length. This is good for privacy, but because of this, it's impossible to measure accuracy on real student answers. An anonymous log with user consent was needed.

In the second part, I'll tell you how I measured the accuracy of all of this and how many times I cheated myself with metrics.