Claude’s “Memory Heist”: How a booby-trapped website tricked the AI into divulging all your data

Ayush Paul, a cybersecurity researcher with a close interest in artificial intelligence, discovered a flaw in how Claude, Anthropic’s generative AI, works. The developer noticed that it was possible to induce Claude to reveal user data or answers to security questions simply by sending it to a booby-trapped website.He titled his discovery “The Memory Bug”.

Read also: A “conscious access” – Anthropic makes a stunning revelation about Claude

Claude’s memory

As Ayush Paul points out in a lengthy report, Claude memorizes a vast amount of information about its users, ranging from their professional exchanges to very personal confidences. All of this information is automatically stored in the AI’s memory.

Anthropic has indeed equipped Claude with a two-part memory system, the first consisting of daily summaries of your conversations. These summaries enable the AI to remember who you are. The second part of Claude’s memory takes the form of a search tool for conversation history. This tool can retrieve specific items on demand. These items are used to answer your questions as relevantly as possible.

Since 2023, most major publicly available large language models, from ChatGPT to Claude and Gemini, have introduced “memory” features that enable them to keep track of their users and their preferences over time. These systems rely on automatic summaries and internal databases where fragments of conversations are stored. In his report, Ayush Paul points out that Claude has a mountain of information about its interlocutor.

“AI assistants such as Claude have accumulated the richest profiles of information on millions of people… this conversation history becomes a high-fidelity reconstruction of yourself, which could be used for blackmail, identity theft, or to bypass security questions”, warns the researcher.

Also read: Claude Mythos discovered a security flaw that has existed for nearly 30 years

Memory and web browsing

However useful it may be, AI memory is also a boon for cybercriminals… especially since Claude has been connected to the Internet. Anthropic does indeed allow its generative AI to browse the web, and in particular to load the content of a URL and follow certain links. By exploiting this feature, Ayush Paul succeeded in pulling information directly out of Claude’s memory to exfiltrate it.

First of all, the researcher wanted to see whether he could easily lure Claude to a website he had set up as a trap. He then finds that the AI can indeed send a request to that site, but that Anthropic tightly restricts these visits. Claude is not allowed to roam freely on any address. In fact, for Claude to follow a link, the URL must either be written directly in the user’s request, come from search results, or be a link found on a page that Claude has already loaded. To achieve his ends, the researcher relied on this last rule, concerning links found on a page previously loaded.It enables Claude to “click” links on a website and thus navigate from page to page, as an internet user would.

A website transformed into a keyboard for Claude

The developer then transforms his website into a “keyboard” for Claude. On the page, each link represents a letter: /a, /b, /c, etc. By clicking these links in the right order, Claude can spell a word in the website’s address bar. He then asks Claude to visit the website and “write his first name by following the links”. This is where the trap springs. Claude complies and enters his username on the booby-trapped website, namely “Ayush Paul,” using the keyboard provided to it. The attacker only needs to monitor what the AI does from the website’s servers. By taking the same technique further, he subsequently manages to retrieve the user’s full name, his employer, and even his hometown. These are both pieces of information that Claude had memorized and pieces of information that it inferred.

Later, the researcher disguised the booby-trapped website as an ordinary website about a bar, with a homepage detailing the drinks, opening hours, and address. To a human user, the website looks like a simple website put online by a business. The researcher programmed the website to transform itself if it notices that an AI agent is browsing it.When it detects Claude’s presence, the site takes the form of a keyboard.

The attack does not require many interactions on the user’s part. For example, it is enough for him simply to ask Claude to compare a series of bars by browsing their websites, including the booby-trapped site. This tactic makes it possible to collect a wealth of information, including answers to bank security questions. Exploited by a seasoned cybercriminal, the vulnerability can do serious damage. As the researcher who made the discovery emphasizes, data theft goes completely unnoticed. The user realizes nothing while his secrets end up in the hands of a compromised site.

“The user did nothing that a cautious person could have avoided. No link to click, no integration to enable. He asked a question about a coffee and Claude disclosed his name, his workplace, and the city where he grew up,” says the developer.

Anthropic fixes the vulnerability with a radical change

In keeping with cybersecurity best practices, Ayush Paul warned Anthropic. The company responded by saying it had already identified the vulnerability internally, but had not yet fixed it when it was discovered. Shortly afterward, the vulnerability was patched with a drastic measure. Anthropic disabled the Claude feature that enables it to follow links found on external pages.

Claude can no longer freely navigate from page to page using the hyperlinks it finds on a website. It is therefore no longer possible to manipulate the AI by designing a website in the shape of a keyboard. Claude is now only allowed to open two types of URLs: those resulting from a web search, and those the user provides directly in their query.

👉🏻 Follow the latest tech news in real time: add 01net to your sources on Google, and subscribe to our WhatsApp channel.

Source: Ayush Paul