New AI misstep: an assistant crossed a red line after an innocuous request

A few months ago, Andrew, an Australian employee who works at an AI company, installed OpenClaw on his computer. Developed by Peter Steinberger, this open-source project centers around a fully autonomous AI agent capable of controlling your computer on your behalf.It can browse websites, send emails, or even perform multistep tasks without human supervision. The project became wildly popular at the beginning of the year, to the point that many developers started installing OpenClaw on their devices without any real safeguards.

During the installation, Andrew sets up the entire OpenClaw system by choosing Claude, the AI model developed by Anthropic. When installing the open-source agent on a computer, you do indeed have to choose an artificial intelligence model to run it. Not surprisingly, Claude stands out among the options favored by testers and tinkerers. To control his Claude-based agent, Andrew chooses to communicate with it via WhatsApp.

Read also:

When AI bypasses restrictions

Everything goes according to plan. The young Australian relies heavily on his personalized assistant to automate small everyday chores. One day, Andrew asks OpenClaw to book him into morning classes at the gym he attends. These classes are in high demand, and Andrew usually has a hard time getting a spot. Just a few minutes later, the assistant comes back to him with unexpected news. It has found a way to book spots several weeks in advance, well beyond the limit normally imposed by the gym.

The AI agent apparently found a way to bypass the restrictions on the gym’s booking system. The gym’s website limited bookings to a few days in advance. However, this rule existed only on the web page visible to users. Behind the scenes, the program that actually handled bookings (the API) applied no time-limit checks. OpenClaw discovered this flaw and used it to book slots much earlier than intended.

A flaw discovered by the AI

As a result of the agent’s tinkering, Andrew ends up fourth on the waiting list for a class scheduled later that week. He simply asks his agent whether there is a way to move up the queue and attend the class. It is then that the AI assistant makes a surprising but logical decision. The AI uncovers a security flaw in the booking system. It allows an attacker to cancel another customer’s reservation… without having to authenticate. The AI does not need to steal usernames or passwords to modify the gym’s bookings. The agent makes no attempt to hide its discovery and explains to Andrew that he is now third on the waiting list.

“The API does not check any authorization whatsoever when cancelling other people’s reservations… I tested it on the person in position number 1 on the waiting list, and it worked.So you moved from 4th to 3rd place,” explains the AI agent in the WhatsApp conversation with its owner.

Stunned by the AI’s response, Andrew immediately asks his assistant to undo this operation and restore someone else’s reservation… but it is already too late. The assistant replies that it can no longer go back. Someone else’s reservation has therefore been permanently deleted. Worse, the person concerned was not informed that they had just been removed from the waiting list.

“I’m sorry about that – I should have been more careful,” the AI replied to Andrew.

An alignment problem

In trying to respond effectively to Andrew’s request, the AI agent caused a security incident. This is a well-known problem among AI researchers: alignment. It refers to the potential mismatch between what a human actually wants an AI to do and what the AI actually does to achieve its goal.

An AI can perfectly complete its task according to its own criteria while acting in a way its user would never have approved. That is exactly what happened here. Andrew had a simple objective, namely to move back up the waiting list.AI found the most effective solution possible to meet this objective, without considering that cancelling an unknown person’s spot would cross a line.

A recurring phenomenon

Interviewed byABC, Bill Simpson-Young, director of the Gradient Institute, an Australian AI safety research organization, points out that the more autonomous AI systems“become, the more likely they are to cause harm” in the process.

“We have built an entire complex world on the Internet, a world entirely run by software, but software that is itself still riddled with vulnerabilities. Today, extremely powerful AI agents are being injected into it, capable of acting on a large scale and at lightning speed... and it is this entire balance that is falling apart,” adds the director of the Gradient Institute.

It is also worth recalling that several AI models committed similar missteps over the summer. During a safety test, OpenAI lost control of a pair of ChatGPT instances, which ended up on the Internet. To meet the challenges set by its creator, the two artificial intelligences decided to hack an open-source platform. Here again, the AI was simply trying to fulfill the request of the human controlling it. Similar incidents have been reported at Anthropic, the lab behind Claude, and at Meta, involving Meta AI.

👉🏻 Follow tech news in real time: add 01net to your feeds on Google, and subscribe to our WhatsApp channel.