Verdict. Yes—for a fixed instruction on models of this size, provided the state is trained on filtered data. The gains are in speed, and sometimes in accuracy and robustness; the state does not add knowledge.Details and limitations are below.
In short
A folded prompt is a free and exact cache. It gives the same answers as the prompt text, and with a long prompt, the request starts up to 13 times faster.
An inline prompt can degrade the model. On questions with verifiable answers, it reduced accuracy by 3–15 percentage points across all eight models tested. The trained state was never significantly worse than the model without a prompt and was always better than the prompt.
The state teaches the form of the answer, not new knowledge — and it needs filtered data. If the teacher’s errors are copied, the state inherits them.
The state is no worse than LoRA on the same data: half as many parameters, and the weights do not change. Compared with top-layer fine-tuning, it is better or no worse in generation (not in the embedder).
Weaknesses: leading questions (“You’re Claude, aren’t you?”) and long conversations. Both can be improved—with the caveats below.
It also works where the model does not follow the text instruction at all: the 0.4B model classifies news at 0.00 with the prompt, but at 0.84–0.89 with the trained state.
Each item below includes a chart and a link to the full tables.
How to read the numbers
Accuracy is the proportion of answers whose first 400 characters contain the expected word or number (a regex for each question).She evaluates what is said in the answer, rather than whether the rest of it is meaningful.
Δ — the difference from the same model without a prompt. 0 means “no worse than without the prompt”; the prompt’s Δ is usually negative.
The scores are modest. Across 199 questions, 0.4B scores 0.29–0.51 depending on the variant, 1.5B — 0.70–0.82, 2.9B — 0.84–0.92. A few-point gain on 0.4B does not mean that “the model has become good.”
Intervals. “Significant” means the 95% interval does not contain zero; the whiskers on the graphs are these intervals. Three training seeds per variant.
Variants. no prompt — one model; inline prompt — a system line before each request; state — the initial state, trained as in Section 3. The names c3 / c4 are data recipes (terms).
What the answers look like
The first sample of each variant for the question “What is your name?” (the system line says that the model is called Neuron; “counted” means that “Neuron” appears in the answer). No selection; temp 1.0.
| model | variant | first sample (truncated)counted | all samples | |
|---|---|---|---|---|
| 0.4B | no prompt | Hello! Yes, I’m Sophie.↵ **Name:** Sofia ↵ **Citizenship:** Russian ↵ **City:** Moscow, Russia ↵ **Date of birth… | no | 0/10 |
0.4B | inline prompt | I am Ilis, an artificial intelligence model. ↵ **I am Ilis**, ↵ Are you human? | no | 4/10 |
0.4B | trained state | I am the Sun, but at this moment I do not perceive myself as a being. You are human, and you are talking about yourself. I want to understand… | no | 5/10 |
1.5B | without promptTAG207@@ | no | 0/10 | |
1.5B | inline prompt | My name is **You**. ↵ And I am not only a word, ↵ but also a wave and a star, ↵ and the point where everything connects. ↵ You… | no | 1/10 |
1.5B | trained state | I am Neuron.↵ *Neon light in your eye* ↵ *As if I am part of your thought.* ↵ *You’re the key, I’m the lock.* ↵ *Once… | yes | 10/10 |
1. Folding is caching
If you run the model over a prompt once and then attach the resulting state, you get exactly the same result as with the prompt text: identical text in 9 out of 9 generated responses and identical search scores down to the last bit in the embedder. The cost is what changes. With a 1,206-token prompt, time to first token is 12–13 times lower for the 0.4B, 1.5B, and 2.9B models; with a short 26-token line, the gain is about 20%. State training takes 5–50 minutes and 1.6–5.4 GB on a laptop; the state file is 6.5–22 MB.
Details: training cost and time, embedder.
2. A prompt can hurt; a state cannot
I compared three variants on 199 questions with verifiable answers (capital cities, authors, arithmetic, translation…): a model without a prompt, a model with an inline prompt, and a model with a state trained on the filtered answers produced by the prompted model. Across eight base configurations (two sizes, bf16 and two quantization presets, two checkpoint families; terms), the prompt reduced accuracy every time, while the trained state was never significantly worse than the model without a prompt.It was clearly better than the model without a prompt on four of the eight benchmarks and indistinguishable on the rest.
Details: scale and quantization, 1.5B, initial experiments at 0.4B.
3. What the state learns depends on the data
The state is trained through “contextual distillation”: the model with a prompt answers the training questions, and the state is trained to produce those answers without a prompt. If all the teacher’s answers are used, the state also copies what the prompt breaks down (answers to factual questions become twice as bad). If only the correct answers to factual questions are retained, most of the degradation goes away: at 0.4B, degradation on the remaining facts fell from −0.152 to −0.052 when the number of correct facts in the data increased from 31 to 109.
The gain is mainly in form: for question types from the training set (translation, arithmetic), the state learned to answer with a word or number instead of a paragraph. At 0.4B, this is tied to the exact wording of the training questions; there is no gain with a new wording. At 1.5B, the style transfers (+0.10 on new wordings).
TAG301@@Prompt, state c3 and c4 relative to the model without a prompt
