After Apex-35B-V1.0 attracted attention for outperforming Flint, the open-source RSI model ecosystem received another impressive result: REMORY achieved fully self-supervised training using only a 1.9B memory module and two H800 GPUs, taking the lead in several key benchmarks; it drove a small model to recursively improve for several days, producing results close to GPT-6.1-Terra.
The paper’s first author, Xia Hanchen, is a researcher at the Shanghai Institute of Artificial Intelligence, working on auto research and agent memory. The corresponding author, Zhu Siyu, is a professor at Fudan University, an AI scientist at the Shanghai Institute of Artificial Intelligence, and a full-time mentor at the Shanghai Innovation Institute.
Paper link: https://arxiv.org/pdf/2610.11287
Hugging Face homepage: https://huggingface.co/papers/2610.11287
GitHub homepage: https://github.com/fudan-generative-vision/Remory
Galaxy platform homepage: https://aistudio.ai4s.com.cn/partner/research-plaza/galaxy-model/model/Qwen3.8-27B-REMORY-1.9B
The recently formed REMORY team—comprising the Shanghai Institute of Artificial Intelligence, the Shanghai Innovation Institute, and Fudan University—delivered leading results on multiple challenging benchmarks with only two core developers, 1.9B trainable parameters, and two H800 GPUs:
Beyond /compact: Let Experience Remain Effective
What is the context window of the large model you use daily? 2M? 1M? Or 0.2M? For an agent dedicated to completing extremely long-running tasks or interacting with users over the long term, context compaction is triggered when its working memory approaches the context-window limit. This is essentially a consensus that every harness must acknowledge. Codex CLI and Claude Code, for example, both have a built-in /compact command. The actual approach is very simple—a summary. The following is the “Generate Summary” prompt behind /compact in Codex CLI.
At this juncture, looking back at the evolution of agents over the past two years, /compact seems sufficient.But is that really the case?
The “sawtooth chart” below depicts the process by which the context continually grew, was compacted, and then grew again while Qwen3.8-27B performed the Core War task in Terminal-Bench 2.1. The red and yellow dots represent tool errors and repeated output, respectively.What REMORY aims to improve is precisely this ability to maintain continuity of memory throughout long-running tasks.
REMORY: Modeling the Memory Residuals Induced by Context Compaction
After a /compact operation, we obtain a summary. The raw context from before compaction and the summary jointly contribute to memory construction. They enter the REMORY model through cross-attention (CrossAttention), which generates a set of residual tokens. These are then appended to the summary so that the main model can continue executing the task.
Here, residual tokens are soft memories represented as continuous vectors that can directly enter the main model's input space. They are generated conditioned on the summary and concatenated with it along the sequence dimension (⊕), which is why we call them the summary's "residual."
Training is fully self-supervised throughout, and the base model weights are frozen. In the first stage, content reconstruction teaches the memory module to retain source information; in the second stage, conditioned on the summary, it learns through KL divergence to continue the subsequent behavior supported by the full history. Gradients pass through the frozen main model, and only approximately 1.9B REMORY parameters are updated.
Tool errors are nearly halved, and generation costs fall by one-third
The benefits of memory are also evident throughout the full execution process. In the Terminal-Bench 2.1 evaluation of Qwen3.8-27B, tool errors fell from 933 to 568, and repeated outputs fell from 459 to 377;the average token-priced generation cost per task fell from $4.29 to $2.86, a decrease of approximately one-third.
In BrowseComp, tool errors fell from 25,244 to 12,698,a reduction of 49.7%; and repeated outputs fell from 25,801 to 18,105, a reduction of 29.8%. Fewer repeated attempts allow the model to spend more of its generation budget on advancing the task.
REMORY's benefits also extend to stronger models and MoE architectures: GLM-5.3-Flash's scores on Terminal-Bench 2.1 and BrowseComp rose from 84.3 and 84.9 to 87.6 and 89.0, respectively. In comparisons where the underlying main models and execution configurations were kept consistent, adding residual memory simultaneously improved scores, reduced errors, and reduced repeated outputs.
Human-on-the-Loop Automated Research
If a benchmark is merely the tutorial level, real-world scientific research is the model's proving ground. To test REMORY's practical performance in scientific work and ultra-long sessions, the team invited several cross-disciplinary scholars from Fudan University to collaborate continuously with a REMORY-powered automated research agent. The researchers provided direction and feedback only through dialogue.
Across sessions spanning multiple days, the REMORY-powered automated research co-worker agent continuously read literature, built experiments, reproduced baselines, analyzed results, improved methods, and revised papers. Meanwhile, the data from each round of context compaction was collected to train the memory module, enabling fully self-supervised iteration and hot updates. Scientific research often spans literature review, hypothesis formulation, code implementation, experimental analysis, and proposal revision. Evidence obtained earlier, approaches already tried, and lessons learned from failure can all influence the next judgment. Whenever an agent crosses a context boundary, it must recover its working state, which can undermine the efficiency and reliability of sustained research. As a general-purpose technology, REMORY carries forward the memory needed for cross-stage tasks within a limited context window. Only by enabling the model to make better use of prior exploration can it continue advancing the research after repeated /compact operations.
