Lightweight LLMs on CPU: What real tasks can be done on an affordable VPS / Habr

models from Hugging Face and most of the registry are in quantization Q4_K_M. Larger Russian models, such as GigaChat 3.1 Lightning, T-lite, and YandexGPT-5-Lite, didn't fit into 4 GB and are waiting for tests on a larger server.

Before the measurements, I had to spend some time with the server and do four things:

• Swap 4 GB with vm.swappiness=10. Without it, the kernel would terminate 4B models on long texts due to memory shortage, but with swap they work, albeit slower.

• Open WebUI was stopped during the measurements: a gigabyte of memory on this server is a quarter of the total RAM.

• fail2ban on SSH. The connections started dropping due to timeouts on the first days of connection: the server was being scanned by bots. When fail2ban began blocking IPs after five failed attempts (with increasing time intervals), the connections stabilized.

• No additional load during the measurements. The trial run was 1.7 times slower when running in parallel with model downloads.

How the experiment was set up

All scenarios were run by a single Python runner without external dependencies, which interacts with the Ollama API. A scenario is a folder containing two files: dataset.json with examples and references, and scenario.py with a prompt, JSON schema for the answer, and an evaluation function.

Datasets. For each scenario, from 4 long documents to 30 short Russian examples with reference answers and difficulty labels. No less than a third of them are "traps": sarcasm, two phones in the letter, a canceled decision during the meeting, "until Friday" instead of a date. The data is synthetic, generated for a fictional electronics internet store with a small sales department.

The same conditions for all models:

• one prompt without customization for a specific model;

• structured response through structured outputs Ollama (JSON schema in the format field), as in real integration;

• temperature 0, seed 42, num_ctx 2048 (for long documents 6144); for Direct ads, temperature 0.7;

• reasoning mode disabled where it can be disabled;

• in memory, one model, first warming up with a third-party request, then all examples one by one, then unload.

The request to the model looks like this:

Metrics. Quality (accuracy, macro-F1, proportion of correct fields or facts), time per task, prompt reading and token generation speed per second, tasks per hour, cold start, and model memory. Swap I evaluated based on the exchange volume from /proc/vmstat, not on the peak of used swap: old pages in swap do not slow down work, only active exchange slows it down.

Manual parsing. Automatic assessment is just the first filter. I read all errors manually and separated model errors from questionable markup and overly strict checks. If the check was at fault, it was corrected, and the recalculation was done using saved responses without rerunning. If the prompt was at fault, the corrected version was tested in a separate run, and both results are shown in the article.

Results by scenarios

In the table below, the best models for each scenario, their accuracy and speed, as well as my conclusions about their suitability for work are listed. Below the table, I analyze each task separately: where models made mistakes and why.

Pattern across all scenarios: models with 2B parameters yielded a working result, models with 1B – no. In the classification task, the pre-trained llama3.2:1b model had 7 errors out of 20. In RAG, for all three questions not covered in the base, it generated its own answers. In data extraction, the 1B models came up with 4-8 fields each.

Classification of customer inquiries

The dataset included 20 customer inquiries across six categories, 8 of which were mixed topics. QVikhr-3-4B, qwen3.5 (2B and 4B), and gemma3:4b all answered correctly, but there was a threefold difference in speed: QVikhr handled 1018 inquiries per hour, while qwen3.5:4b handled 312. The reason was not in generation (answer – 7-8 tokens), but in reading the prompt: QVikhr reused the already read system prompt (prefix cache) and processed at 154 tokens per second, whereas the hybrid models qwen3.5, granite, and gemma3 with a sliding window each read the prompt anew at 28-46 tokens per second. On CPU, this is the main factor affecting speed. In a complex example "the seller doesn't know about the product, train the staff," 4 out of 9 models classified the complaint as a product question.

Sentiment of reviews

Models were given 24 reviews, including mixed, sarcastic, and slang-laden ones. QVikhr and qwen3.5:4b handled 22 out of 24, but QVikhr did so twice as fast. Almost no model recognized sarcasm. The review "Thank you for delivering it within three weeks, I almost forgot I had ordered it" was rated as positive or neutral by 7 out of 8 models, while QVikhr recorded the reason as "fast delivery within three weeks." All 1B models labeled all mixed reviews as negative and sometimes switched to mixed languages ("kurьер"). In practice, you can do this: if the model sees positive, but the star rating is low, the review goes to a person for verification.

Extracting data from requests

The input was a dataset of 10 emails to the sales department, each with six fields: contact, company, phone, email, budget, and deadline. Some emails contained "traps": budget ranges, INN and account numbers to confuse, the name of the day of the week instead of a specific date, etc. Contacts and budget were almost perfectly extracted by strong models, but deadlines were not. None of the models translated "until Friday" from a Tuesday 20.10 email into 23.10. The first version of the prompt sometimes confused the models. After clarifying the rules, qwen3.5:4b extracted 59 out of 60 fields with no fabricated data, but gemma3:4b got worse after the same corrections (57 → 54). Rules for small models should be checked separately on each.

Summarization

For this scenario, I prepared four documents, including a 2000-word meeting transcript where decisions changed during the meeting. The task was to write a summary of five main points for the manager. Strong models correctly captured the final decisions, not the initial ones. However, distortions appeared in the summaries, which only a person can detect: gemma3:4b calculated a 13% sales increase instead of 8%, and qwen3.5:2b wrote that technically complex products "are not returnable", losing the note "except for defects". Only qwen3.5:4b managed without distortions, but it took 71 seconds to process a 600-word document, and the server heavily used swap during this time. QVikhr, the leader in short tasks, turned out to be the slowest here: 6.3 minutes for the transcript. She has normal attention, and the context cache grows with the length of the text, while for hybrid qwen3.5 it almost does not grow.

Knowledge base answers

The database included five documents (about 4700 words), split into 34 fragments. The model bge-m3 converted the fragments and questions into vectors. Four most semantically similar fragments were selected for each question, and only they, not the entire database, were included in the prompt. Indexing the entire database took 49 seconds, search took 0.1 seconds, and the relevant document was always found. QVikhr, qwen3.5 (2B and 4B), and granite answered correctly on 14 out of 15 questions and honestly wrote "no information in the database" when there was no answer. A weakness was rule exceptions. For the question "Can a working laptop be returned?" the models answered "Yes, within 14 days," although the fragment retrieved directly implied the opposite. However, this can be fixed on the database side: exceptions should be formatted as explicit question-answer pairs. The speed of 40-95 seconds per question is acceptable for an internal assistant, but not for customer chat.

Yandex Direct Advertisements

The models received six briefs. For each brief, five advertisements were required: a title up to 56 characters, text up to 81 characters, only facts from the brief.Small models do not know how to count characters.The length of the text for most of them was in the range of 80-170 characters, while the limit was 81. Only qwen3.5:4b fit within the limit in all 30 advertisements. Only 50% of the generated advertisements required factual corrections. Qwen provided fictional discounts, and gemma3:4b invented a loan with "bank solution in 5 minutes." QVikhr provided almost identical options. Working scheme in such circumstances: generate 10–15 variants and filter them by length and digits from the brief.

CRM data normalization

Each model received three queries, ten "dirty" lines of each type: phones, company names or addresses ("мск садовая 14-8", "ебург"). Strong models handled addresses well – exactly such cases are hard to cover with regular expressions. However, there were problems with phone numbers: even strong models erased numbers and converted Belarusian ones into Russian. Such type of tasks is better to be passed to the code. The most dangerous defect found – line shifting. In one of the queries, answers from 4 out of 9 models shifted by one position, and each record was assigned to the wrong client. In the live integration, the answer should contain the line id, and the code should check the match.

Load testing: how much can the server withstand

In the scenarios above, the queries went one by one, so the measurements show how long one user waits for the response. In practice, queries come simultaneously from several employees or integrations, and it is important to know how many queries the server can handle and how much the delay will increase. To measure this, I wrote a small load script: it sends a series of queries in 1, 2 and 4 threads and calculates the throughput, median and p95 of the delay, memory and swap exchange. The queries were of two types: short ones, like in the classification of calls, and long ones, like in the knowledge base responses (about 2 thousand. Tokens on input).

Three Ollama configurations:

• A – Ollama settings from the template without changes, one request at a time;

• B – OLLAMA_NUM_PARALLEL=4;

• C – B plus OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0.

And comparison of configurations on QVikhr - the only one among the leaders that can handle parallel requests:

It follows that:

• One model – one queue. Throughput on 1, 2, and 4 threads is the same (qwen3.5:4b: 272 / 279 / 274 per hour), only the waiting time increases. Four employees asking questions simultaneously wait four times longer.

• Parallelism on 4 vCPU doesn't help. The processor simply splits between requests: generation per request drops from 6.9 to 1.2 tokens/sec, and slots reserve memory for the context of four requests, causing the server to swap on long tasks.

• qwen3.5 doesn't support parallel requests at all. Ollama 0.35 writes to the log "model architecture does not currently support parallel requests" and silently processes them one by one, while still reserving memory for slots.

• Flash attention harms this CPU: prompt reading drops from 27 to 12 tokens/sec. Compression of KV-cache without Flash attention is not enabled in Ollama.

• Long context hits memory limits. When there's a queue of 2-4 long requests to the 4B model, the kernel kills the process (Out of memory: Killed process ... (llama-server)), the client receives HTTP 500, and the model reloads approximately 30 seconds. Only qwen3.5:2b survived under load without errors.

In the load test, Ollama performed best, with no changes made to its settings (config A). Parallel slots, flash attention, and KV-cache compression on this server did not yield any improvement.

Issues with measurements and how to overcome them

Some measurements had to be redone. If you repeat the experiment, keep the following in mind.

1. Prompt cache distorts measurements. In the first load test, long queries were read at a speed of 1900 tokens/sec instead of 27: the same questions were repeated, and llama.cpp took them from the cache. The results had to be discarded. The solution is to add a random tag to each query and warm up with irrelevant text. For the same reason, the time of the first example was underestimated in early scenarios.

2. Non-disabling reasoning nullifies results. LFM2.5-2.6B ignored think: false, spent all the token limit on reasoning, and returned an empty response. In the first classification run, it failed to label any of the 20 queries, and the issue was due to the token limit, not the model's quality. With a token limit of 1024, it responded adequately, but 10 times slower than others. Reasoning on CPU did not pay off in any scenario.

3. Too strict auto-checking lowers scores. The first version of the evaluation script counted an address only if it contained «г.» before the city name and «д.» before the house number. Models wrote «Москва, ул. Ленина, 5»: semantically correct, but the script gave them all zero. Similarly, the check for numbers in the ads considered «15 тыс.» as fabrication, because it looked for an exact number from the brief in the text. Therefore, errors should be handled manually, and raw answers should be saved to recalculate the score without rerunning.

4. Examples in the prompt should not match the tasks from the test, otherwise you are measuring the model against the hint.

5. External load distorts the results. Downloading the model in parallel with the run slowed it down by 1.7 times. For the same reason, the response speed on VPS may vary from run to run due to the activity of neighboring hosts on the same server.

6. Bots break SSH connections. Servers with password login start brute-forcing immediately. fail2ban is a hygiene minimum here, but key-based login is more secure.

7. Switching between models consumes up to 35 seconds. Two 4B models in 4 GB do not fit; each switch takes 20-35 seconds of loading time. In production, the model needs to be kept loaded. In the template, the model is unloaded after 30 minutes of inactivity (OLLAMA_KEEP_ALIVE=30m), and the first request after the pause again waits for the model to be reloaded.

What in the end can be trusted to the model on the server for 1,500. Even on a VPS without GPU, local LLMs perform well on some common business tasks.

They can generally be trusted to:

• categorize incoming messages and determine the sentiment of reviews – up to several thousand per day;

• extract contacts and budgets from applications for CRM (with human confirmation);

• answer employees' queries from internal knowledge base in under a minute;

• at night, summarize documents, draft advertisements, and clean up the customer database.

What you shouldn't do, is to put such a server in a chat with customers, trust the model with arithmetic calculations, record responses in CRM and marketing office without checking, and rely on several simultaneous long requests.

Which model to choose

Lightweight models understand the meaning of text well but are bad at calculations and don't check themselves. Therefore, for all scenarios, the optimal scheme would be one where the model handles tasks requiring text understanding, the code performs formal checks (dates, phone numbers, string length, numbers from the source), and a person confirms what goes outside.

Set up a queue with a single handler, a timeout of 5 minutes for long tasks, and retry on 500 errors. In RAG, try using three fragments instead of four: in my test, the needed document was always in the first fragment, and a short prompt is read faster.

The bottleneck is memory. With 8 GB RAM, 4B models will stop going into swap and crashing on long texts, and two models and a chat can fit in memory. At Justhost.ru, the difference between 4 and 8 GB with the same 4 vCPU is about 550 ₽ per month. More cores will speed up reading long texts, but probably won't give a multiple increase in speed. A chat for clients with a response in seconds is already a task for a GPU with a different cost order.

P.S. If you have run lightweight models on CPU for your own tasks, write in the comments which scenarios and models worked for you.