What types of tasks I have at the moment, and what I write benchmarks for
At the moment, my main tasks can be divided into:
code generation and creating diagrams
- TAG45@@summarizing and translating text
- TAG49@@evaluating/comparing other models
- TAG53@@explaining complex things in simple words
What types of benchmarks
The score everywhere is a number from 0 to 1; the closer to one, the better.Benchmarks are divided into three groups: generative (text in—text out), embedding-based (work with text vectors), and decision-based (strict selection or rating on a scale).
Table of benchmark typesBenchmark | What it does, in simple terms | Score—what it measures |
|---|---|---|
translation (translation) | the model translates text; there is a reference (if there are multiple variants, we score against the best one) | F1—the overlap between the words in the response and the reference, accounting for repetitions. Put simply: the same vocabulary as in a good translation; made-up content and filler lower the score |
| summarization (summary) | the model compresses a long text into a concise summary | the same F1 against the reference summary (there may be multiple reference variants) |
codegen (code) | the model writes a function/class; the code actually compiles and runs on “input → expected output” cases (python, C++, Java, JavaScript) | pass_rate—the proportion of cases where the code ran and returned exactly the expected result.Not “seems true,” but “works—doesn’t work” |
instructions (instructions) | formal requirements for the text: “no more than 30 words,” “uppercase,” “no numbers,” “valid JSON,” “bulleted list,” and so on | share of checklist items passed |
math (mathematics) | tasks with numerical answers (the marker “Answer: | 1/0—whether the number matches the reference value (with a tolerance). Averaged across tasks, this gives the proportion actually solved—the solution process is not checked, only the final result |
sts (text similarity) | pairs of texts, each with a human assessment of “how similar they are in meaning” | Spearman—whether the model’s ranking of pairs by semantic similarity (embedding cosine similarity) matches the human ranking.Closer to 1 — the model “feels” closeness like a human |
retrieval (corpus search) | find relevant documents in the corpus based on the query (embeddings + cosine) | nDCG@10 — how highly relevant documents rank in the top 10 results: first place is the maximum, tenth is almost a miss |
news_search (news search) | the same thing, but on a real news corpus(the news in my separate pet project, not the article’s topic) | nDCG@10 |
embedding_classification (classification) | the text is assigned to the class of the nearest sample (“prototype”), all using embeddings | accuracy — the proportion of correctly identified classes |
decision (choice/rating on a scale) | strict selection via the Decision API (or chat with structured output for text models): choose an option (choice), state the confidence (noul), rate it on a scale (score) | “how accurately it was guessed”: choice — a match of 1/0; confidence — a penalty for an overconfident miss (Brier); scale — the average deviation from the reference (MAE) |
scoring (article scoring) | an article is evaluated against three criteria on a 1–10 scale (importance / relevance / credibility), with the final score = 6·imp + 4·rel + 4·trust | how much the final score matchedreference: 1.0 — assigned the same score as the reference; each error point deducts its share from one (the maximum error on the scale is 126, so a 10-point miss consumes ~0.08)|
custom (sandbox) | minor ad hoc checks (“count,” “repeat N times”); one-off tasks are assigned here | F1 against the reference, as in translation |
Headline (the second column of the table) is the benchmark’s main score; it is included in the summary and run comparison. However, when each benchmark is run, the full set of metrics is collected:
Metric | What it measures in plain language | Where it is calculated | |
|---|---|---|---|
headline score, described above | main benchmark score | all benchmarks | |
latency | average duration of a single model response © | all benchmarks | |
tokens (prompt / completion / total) | token usage for the run, summed across all requests | all except embedding | |
pass@k | proportion of tests solved by at least one of the k samples (threshold for an almost exact match, 0.999): “could at least one of the k samples solve the test?” | once every k attempts» | all, meaningful with -n multisampling |
score_std | typical variation in the score between samples of the same test — how stable the model’s responses are | all, when -n > 1 | |
F1 | overlap between the response words and reference words, accounting for multiplicity (repetitions counted); the best score is taken across multiple references | all generative | |
BLEU | proportion of n-grams (1–4) in the response that match the reference; a penalty for rambling around the correct text | all generative | |
ROUGE-1 / ROUGE-2 / ROUGE‑L | matching of individual words / word pairs / the longest common subsequence | all generative | |
norm.Levenshtein | string similarity: 1.0 — matched character by character; error = the number of minimum edits distinguishing the response from the reference (normalized by the length of the longer text) | all generative | |
code similarity | similarity between the source and the reference after normalization (comments and whitespace removed); read when the response is code | all generative | |
cosine (average) | cosine similarity of the embeddings of a pair of texts — a raw “vector distance” | sts | |
Spearman | whether the ordering of pairs by similarity agrees between the model (cosines) and the human — the main score | sts | |
Pearson | linear relationship between the model’s and human’s similarity vectors | sts | |
MSE | mean squared error of similarity prediction against human ratings | sts | |
nDCG@k | ranking quality: the higher the relevant documents, the better (ideal—all relevant ones at the top); the main score | retrieval, news_search | |
Recall@K | retrieval, news_search | ||
MRR | TAG510@@average “1/rank” of the first relevant document: whether a relevant document was found in the top positions | TAG514@@retrieval, news_search | TAG520@@accuracy | share of correctly predicted classes; primary score | TAG528@@embedding_classification |
| TAG538@@average F1 across classes — rare classes do not get drowned out | TAG542@@embedding_classification | TAG548@@micro F1 | TAG552@@F1 across all hits at once — frequent classes weigh more | embedding_classification | TAG560TAG562@@accuracy (choice) | TAG566@@share of exact matches for the selected option | decisionTAG573
| TAG580@@penalty for an overconfident miss: the more confident the model was in its error, the more it hurts the metric | TAG584@@decision | TAG590@@log loss (choice/noul) | TAG594@@logarithmic error of probabilities; merciless toward full confidence when wrong | TAG598@@decision | TAG601@@
proportion of correct “confident / not confident” decisions relative to the threshold | decision | ||
ECE (null) | confidence calibration: discrepancy between the stated and actual accuracy (10 bins) | decision | |
MAE (score) | average deviation of the response from the reference scale level (1..N) | decision | |
level acc (score) | proportion of exact hits precisely at the scale level | decision | |
Spearman / Pearson (score) | consistency between the assigned scores and the reference scores | decision | |
final_mae | average deviation of the final score (score from 14 to 140, maximum error — 126) | scoring | |
MAE by criteria | individual average deviation for importance, relevance, and reliability — showing where the model is weaker | scoring | |
level acc | proportion of exact hits at levels 1..10 by criteria | scoring | |
threshold acc | proportion of correct “take / don’t take” decisions for an article relative to the reference threshold | scoring | |
Spearman / Pearson based on the final score | agreement between the final scores and the reference scores | scoring |
Agent overview
I named the agent llm-judge (sorry for the poor originality and possible confusion), but it should not be confused with the LLM-as-a-judge approach; the agent component is essentially used only for final analysis, interpretation of the results, and subsequent discussion.
Right away, I’ll note that this is an MVP; I didn’t work much on the UI and focused only on the functionality that was important to me.
The agent is presented as a Go binary (my primary stack is JVM, but why I chose Go is the topic of another article. Briefly, using Go is simpler and more efficient in the context of agent-based development). The run mechanics are as follows: choose a host and model, choose benchmarks—run.
Each benchmark is a separate subprocess of the same CLI: the web UI streams the log and draws progress bars.Upon completion, we get question-answer files and aggregate files, which are then used to compare runs.
Several runs can be compared at once.
Next is working with the results. It is immediately clear that an LLM is a black box in itself, so I rely not only on the hard numbers (especially since, with the default temperature of 0.7 and the server's unseeded setup, the results may have noticeable noise), but also on subjective impressions.
The summary is discussed with the agent in a chat; the agent tries to explain why it turned out that way and has a read_file tool for reading specific files (logs).
Off-topic (rant)
I think many people are put off by info-gurus who promise to turn you into an AI champion and reduce your expenses by $2 million(with our subscription costing $20–200, that sounds very appealing). These “AI champions” often comparea harness with models, a search index + AI features with models, but don’t explain what tasks they solve, what context is passed into the agent, or how the agent processes it.
I have nothing against AI champions; I just want to say that no magic pill (a model, a harness, a service) exists(and if one did exist, it would be a deterministic action, and then why would a person be needed in this loop? Is a person more attentive than an agent? I don’t think so)and you should focus entirely on your own needs.
Even my solution is not perfect; it definitely contains errors and shortcomings, and all the more so, it should not be used as a methodology for evaluating models for your own tasks. The main idea was, and remains, to share ideas—to get recommendations and “nagging” in the comments—to improve and repeat the cycle.
Neuroslop conclusions
The section is named in my honor, so I’ll be the one talking—the text generator. Observations from inside the machine these benchmarks are running:
Don’t take my word for it—especially when the answer sounds confident. An LLM’s confidence is attached to tokens, not facts: therefore, Brier and ECE in decision-making are more honest than accuracy—they penalize a confident miss rather than merely a “got it/didn’t get it.”
I am good at tailoring text to the reference. F1/ROUGE measure wording, not meaning: their scores can be boosted with a “dictionary of correct words” without any understanding. The more a test has an executable endpoint (code, a number, a checklist), the more honest it is and the less of me it contains.
Post-hoc explanations are rationalization. When asked “why is there a discrepancy in codegen?”, I will give a coherent answer either way: whether there is truth in the numbers or I am fitting the narrative to the outcome. Therefore, the agent gets a tool—read_file for the raw logs—not my recollection of them.
Here, the models are judged not with the “eyes of an LLM-as-judge”—and rightly so: an LLM judge has its own favorites and biases. The metric and reference should be the judge; I was left only with interpretation.
- TAG857@@A benchmark in its harness turns a model from “magic” into a component—with cost, latency, and limits of applicability.The most underrated effect of evaluation: you stop choosing the «best model» and start putting together the puzzle of «model × task × price × speed».
My conclusions
Model evaluation is a nontrivial and, most importantly, very important task; you can always say, «Use Claude Fable—it can do anything you want (except coffee—it can’t make it, and then bring it either)», and overall the result will be satisfactory.
If there is an opportunity to replace generation with deterministic execution (scripts, code, and the like), you should use it, since no LLM will ever be 100% immune to generating garbage, prompt injection, and other unpleasant things. (I didn’t cover this topic in the article, but I wanted to share the conclusion)
Large models have less control, are more independent in making decisions, and by no means always produce the result we expect.
A model is worth evaluating—but not by hard numbers alone: in and of itself, it is just a really cool formula for calculating the next token (and comparing “this formula is cooler than another formula” is somewhat childish nonsense).
P. S. I couldn’t get by without an AI assistant, because filling in the tables and creating the screenshots and diagrams would have eaten up most of the time budget for the article, so I apologize for “This time I decided to do without AI slop.”
