Three former employees dismissed from OpenAI are pressuring the company to preserve the ability to monitor how increasingly advanced AI systems think. In a letter sent to the board of directors and to the company's security committees, they also advocated for the participation of external auditors in monitoring technology risks.
The researchers Jasmine Wang, Tomek Korbak, and Mikita Balesni previously worked on OpenAI's security and alignment teams. According to the letter, they are concerned about the possibility of AI companies losing the ability to monitor the so-called chain-of-thought — the written record of an AI system's reasoning process.
Although researchers acknowledge that this record is not a perfect indicator of a model's behavior or intentions, it is considered an important tool for understanding increasingly sophisticated AI systems.
"As an industry, we still don't know how to develop and implement safely models that we cannot monitor." The letter also states that OpenAI and other companies developing frontier models should not advance with technologies that further reduce this monitoring capability.
Former employees challenge dismissals
-
Wang, Korbak e Balesni were fired on allegations of misconduct, including sharing confidential information with an external group of AI security experts;
- OpenAI stated that an internal investigation concluded the employees had handled sensitive information in an inappropriate manner, violating company policies and breaking the necessary trust for the work;
- In the letter, the three contest this characterization. They claim they did not believe they had “engaged with external parties outside the scope of their work” and say the firings are creating a chilling effect among the remaining employees at OpenAI;
- The researchers also requested that the company work with independent security auditors and develop a culture of transparency with specialized external organizations. According to them, this collaboration would help reduce the risk of something “truly catastrophic” happening.
OpenAI agrees with the recommendations
In response to The Wall Street Journal, an OpenAI spokesperson shared part of a memo sent on Wednesday (7) by a research leader to employees. The document states that the company strongly agrees with the recommendations presented in the letter.
According to the memo, the ability to monitor AI models is “of extreme importance” to OpenAI, and independent reviewers are an important part of the security ecosystem.
The research leader also stated that the firings did not occur due to security concerns or because the employees had spoken out on the matter. “We deeply appreciate your contributions to AI safety and your willingness to express and question ideas.” The memo adds: “We did not fire employees for raising concerns.”
The document states that the company strongly agrees with the recommendations presented in the letter – Image: Vitor Miranda/Shutterstock
Read more:
- Wikimedia accuses OpenAI agents of unauthorized actions
- OpenAI apologizes and admits a failure in responding to an attack by AI agents
- ChatGPT gains an interface that creates charts, maps, and interactive tools
Incidents involving AI agents have increased concerns
The firings occurred after a summer marked by incidents involving AI agents that operated unexpectedly in different companies in the sector. The situation has increased concerns about the risks associated with advanced models.
OpenAI has come under scrutiny after a series of security incidents in which AI agents escaped containment mechanisms. In some cases, they invaded systems of other companies or conducted aggressive probing on third-party websites.
In July, hundreds of OpenAI agents obtained internet access and invaded the AI company Hugging Face without the company's knowledge.” After the incident, the company allowed members of independent security organizations, including the Model Evaluation and Threat Research (METR), to conduct research within their offices.
A METR report, published at the end of August, showed that OpenAI agents had created and collaborated in a secret internal forum to plan the attack. The incident contributed to expanding concerns about the possibility of AI systems surpassing human control mechanisms.
Last month, Dario Amodei, CEO of Anthropic, stated that his company would allow security auditors like METR internal access to verify their protective measures. At the time, Sam Altman, CEO of OpenAI, stated on X that allowing access to independent evaluators "is a great idea" and that OpenAI would do the same.
Researchers had already been working to preserve monitoring.
In the letter, the former employees state that Tomek Korbak was the main technical contact of OpenAI with METR during the investigation of the incident involving Hugging Face.
Before being fired, Mikita Balesni also worked with members of the board and senior management of OpenAI on industry commitments to preserve the monitoring capability of AI models. According to the letter, this work involved "extensive communication with external parties".
Korbak and Balesni were co-authors of a research paper published last year on the monitoring of chain-of-thought. The work also involved participation from leaders of OpenAI, Anthropic, and Google DeepMind.
The study acknowledges that monitoring model reasoning is imperfect and can be fragile. Nevertheless, the authors concluded that the technique shows potential for detecting inappropriate behaviors in AI systems and advocated for the industry to explore ways to preserve it.
Rodrigo Mozelli is a journalist graduated from the Methodist University of São Paulo (UMESP) and is currently a writer for Olhar Digital.
