"Who said just now that they plan to plant flowers?"
For humans, recalling the previous conversation and context often allows them to answer. However, for an AI that has watched hours of videos, this question is not straightforward: it may have found flowers, found several names, and found discussions about planting, yet still cannot determine who proposed the plan.
Clues are found, but relationships are not remembered.
This highlights a key issue for long-term memory in multimodal agents: memory should not merely be a simple storage of past information, but also understand the relationships between people, objects, actions, and events in the experiences.
Especially when AI assistants begin to accompany users for long periods, the value of memory is no longer limited to answering a specific question. It also needs to gradually understand who the user is, what they have done, what they care about, and their preferences, and connect new experiences from each interaction with existing memories, continuously accumulating, refining, and solidifying them during prolonged use, ultimately forming truly personalized memories. This is also an important foundation for agents to achieve continuous learning and self-evolution (RSI).
Around this issue, a research team from Zhejiang University and Lenovo Group collaborated to propose EM²Mem (Event-Centric Multimodal Memory), which constructs multimodal long-term memory centered on events. It no longer stores video frames, voice dialogues, and text information in isolation, but during the memory construction phase, it associates the people, objects, actions, and temporal clues within the same event, allowing the AI not only to find out what happened, but also to understand who it relates to and how the experiences are connected.
Figure 1 | EM²Mem constructs multimodal long-term memory centered on events.
-
Paper: EM²Mem: Event-Centric Multimodal Memory for Large Language Models
-
Conference: EMNLP 2026 Findings
-
Paper link: https://arxiv.org/abs/2609.00551
-
Code has been integrated into: https://github.com/zjunlp/LightMem-Ego
01｜Why finding relevant content doesn't equal remembering?
Evidence in long videos is often scattered and cross-modal.
A person's name may appear in the dialogue; what they are holding needs to be seen in the video; whether a task is completed requires combining several previous experiences to determine. However, the questions users ask are usually complete: "Who proposed this plan?" "What happened later to this event?" "What do I usually do after arriving at the office?"
Many existing methods process videos into descriptions, transcripts, key frames, or graph facts, storing them separately, and then retrieve relevant content when answering questions. While this approach can find clues, the model still needs to temporarily judge: are these clues referring to the same event? Who is related to whom? What is the sequence?
The paper summarizes this bottleneck as retrieve-then-align: first retrieve, then align .
If a piece of evidence is not sufficiently relevant on its own, it might even be filtered out during the retrieval phase, losing the opportunity to jointly explain the question with another clue.
EM²Mem changes this order: align-then-retrieve, first align, then retrieve. It moves the organization of evidence to the memory construction phase, allowing subsequent question-answering to directly revolve around events.
02｜Establishing an event archive for a piece of experience
One can understand the event memory units in EM²Mem as an event archive with a time address. The system first divides long videos into shorter base segments, with a typical granularity of 30 seconds in the paper. Each segment corresponds to an event anchor, recording its identity and start/end time, and attaching relevant evidence to the same address.
This anchor provides a common index: images and dialogues can point to the same experience, rather than being stitched together solely because of similar keywords.The first layer: remember what happened at that time
.Each event unit retains key frames, visual descriptions, dialogue transcriptions, as well as structured fields such as actions, objects, topics, scenes, and entities. For example, a discussion around a floral arrangement records who said what, as well as the flowers and vases in the画面. The second layer: retain the cause and effect .
Looking at just 30 seconds may not be enough to understand the whole event. EM²Mem therefore builds multi-scale contexts such as 3 minutes, 10 minutes, and 1 hour, linking longer-range textual and visual summaries back to the base events, maintaining the connection between local details and longer narratives.The third layer: connect different experiences and accumulate long-term patterns
.The plot graph connects people, objects, locations, topics, and time relationships within specific events, while the semantic graph further organizes habits, preferences, daily routines, and stable relationships. These high-level information still links back to the events that support them, making it easy to trace the basis. Figure 2｜From left to right: building event memory units, connecting experiences via plot and semantic graphs, then retrieving, expanding, and filtering evidence based on the question. Zoom in for mobile reading. When a question comes in, the system first locates the relevant events, then supplements clues along time or entity relationships, retains necessary evidence through a selector, and forms a compact answer context. For questions that require visual verification, a few key frames can also be added.
This allows the model to simultaneously obtain:what happened, who it involved, and where the evidence came from.
The third layer: connect different experiences and accumulate long-term patterns. The plot graph connects people, objects, locations, topics, and time relationships within specific events, while the semantic graph further organizes habits, preferences, daily routines, and stable relationships. These high-level information still links back to the events that support them, making it easy to trace the basis. Figure 3｜Similarly, clues are found around flowers, but the key difference is whether the connection between people and plans can be preserved. This figure shows the qualitative case study in the paper.
This case makes the motivation of the paper very intuitive: when facing multiple people, multiple objects, and multiple time segments, in addition to the relevance of the retrieved content, the memory organization needs to preserve internal relationships within events.The fourth layer: experiments: answering more accurately, and evidence is located better
.
The study evaluates EM²Mem on three long video multiple-choice question answering benchmarks, covering first-person daily experiences, ultra-long first-person reasoning, and open-domain long video understanding.
† The results of WorldMM reproduced under the same evaluation setup as EM²Mem; the Ego-R1 Bench baseline uses the original reported results from WorldMM's paper. The original paper reports an average accuracy of 65.6% on EgoLifeQA and 76.6% on Video-MME (L). In addition to the final answer, the paper also checks whether the system finds the corresponding evidence snippets.
In the strict 30-second event-level evidence recall evaluation, only the correctly annotated event snippets are considered correct. EM²Mem's Top-5 recall rate reaches 30.8%, which is 7.0 percentage points higher than WorldMM's five-round iterative retrieval of 23.8% ; the Top-1 recall rate also reaches 23.0%.
Figure 4 | Event-level evidence recall. The x-axis represents the number of Top-k events returned for EM²Mem and the maximum number of k rounds of retrieval for WorldMM, which are different budget metrics.
These results indicate that event memory is easier to retrieve accurately, making the quality of memory beyond accuracy checkable.
05 | Why Structured Information and Early Alignment are Effective
Where does the performance improvement come from? The paper further compares two aspects: the form in which visual information enters memory, and when multimodal information completes alignment.
In the analysis experiment on the first 250 questions of EgoLifeQA, the study compared raw frames, flattened descriptions, and structured event fields. The results show that the combination of structured event fields + alignment during construction achieves the highest accuracy of 71.2%; similarly using structured fields but fusing evidence during retrieval yields an accuracy of 68.0%.
Figure 5 | Combining structured event fields with alignment during construction achieves the best results in this set of analysis experiments. The evaluation here is on a subset of the first 250 questions, differing from the complete baseline results mentioned earlier.
Structured fields explicitly retain clues such as characters, actions, and objects, while early alignment allows these clues to have a shared event context before retrieval. Combining both makes it easier to convert multimodal observations into retrievable and answerable evidence.
06 | Preparing the Organization Work in Advance, Making Repeated Question-Answering Lighter
In the inference efficiency comparison for EgoLifeQA, EM²Mem compared to WorldMM:
-
The average time per question-answer pair drops from 459.00 seconds to 98.21 seconds, approximately 4.67 times faster.
-
The total token usage during the inference phase in the evaluation drops from 42.03M to 15.27M, a reduction of 63.66%.
These gains come from the pre-organized event memory: during questioning, relevant evidence can be read and filtered, reducing the need for temporary organization of multimodal clues.
More importantly, memory construction is done once, and subsequent queries can continuously reuse it. As the number of questions increases, the construction overhead is gradually amortized, and the lower per-query cost will further translate into end-to-end advantages. In the paper's EgoLifeQA evaluation, after including memory construction and 500 question-answer pairs, the end-to-end total time drops from WorldMM's 318,210 seconds to 105,160 seconds, approximately 3.03 times faster .
The appendix further analyzes the relationship between the number of queries and cumulative costs. Based on the actual evaluation execution process of the paper, approximately 24 queries can offset the additional construction time . When counting input and output tokens equally, after approximately 536 queries, the total token usage including the construction phase also begins to be lower than WorldMM . When continuing to reuse the same memory, the end-to-end average time and token usage per query can further demonstrate the advantages of EM²Mem.
Figure 6 | Cumulative cost of memory construction and subsequent Q&A. The bar chart corresponds to total time, the line chart corresponds to total token usage, and vertical dashed lines indicate the intersection points of the two costs. After surpassing the respective intersection points, the cumulative cost and the cost per query of EM²Mem are both lower than WorldMM. Time is calculated based on the paper's actual evaluation execution process, and tokens are counted equally by quantity.
This aligns with the usage pattern of long-term memory. Personal experiences, video archives, or corporate knowledge are typically queried multiple times, and pre-organized event memories can demonstrate value through continuous use. When encountering questions requiring verification of visual details such as colors or small objects, the system can return to key frames for validation.
From Memory Research to Application: The Multimodal Agent Memory System LightMem-Ego
Based on EM²Mem, the team further developed the multimodal agent memory system LightMem-Ego and provided two application cases to integrate multimodal long-term memory into daily life and scientific research work.
Personal Life Assistant: Let AI remember the点滴 of daily life. By recording daily experiences through mobile phones and smart glasses, AI can help users find misplaced items, recall past conversations, organize life notes, and create personalized assistants.
Scientific Laboratory Assistant: Let AI remember the experimental process. By continuously recording videos, operations, and conversations during experiments, AI can assist researchers in retracing experimental steps, locating key operations and phenomena, organizing experimental records, and providing support for scientific research.
-
LightMem-Ego Technical Report: https://arxiv.org/abs/2607.11487
-
View system technical details: https://github.com/zjunlp/LightMem-Ego
LightMem-Ego was jointly developed by students including Chen Yijun, Xiao Boyi, Zhao Yixian, He Yi, Xia Haoting, Guo Xiangyu, Xu Buqiang, etc.
