OpenAI and Anthropic are negotiating with independent evaluators over what information can be disclosed after safety testing of their artificial intelligence models. The standoff involves protecting trade secrets and ensuring transparency about possible risks of the technology.
According to Reuters, the companies sought to control publication of the reports, raising questions about the independence of the experts responsible for evaluating increasingly powerful systems.
TAG32@@External experts evaluate AI models, but access to the labs may influence what reaches the public. – Image: JRdes/Shutterstock
What the companies want to control
The tests include external teams that work temporarily inside AI labs.This access facilitates the analysis of models, but also creates a conflict: confidential information may be necessary to identify flaws and assess risks.
Among the measures discussed are the presence of laboratory staff during evaluators’ work and restrictions on the same team working simultaneously for competing companies.
Anthropic states that, in its current engagements with evaluators it has already worked with, the specialists control the methods, criteria, conclusions, and published content. For new teams, however, it reserves the right to comment on and approve reports while establishing the working relationship.
OpenAI denies seeking approval of the full text of the reports. According to the company, confidential information may be redacted, and evaluators can indicate whether there have been significant cuts.
Independence under pressure
Without common standards or specific rules, laboratories can influence evaluations by controlling access to systems. Marc Warner, chief executive of Faculty, a company affiliated with Accenture that was selected by Anthropic as its first embedded evaluator, emphasized the importance of trust in the results.
Evaluators are only useful if people can trust what they report, so independence and transparency are fundamental.
Marc Warner, CEO of Faculty, to Reuters.
Rumman Chowdhury, the former U.S. envoy for AI in the Joe Biden administration and CEO of Humane Intelligence, also warned about the pressure on researchers.
“Evaluators have this privileged access, but, explicitly or implicitly, they know it comes at a cost,” she told Reuters.
Technical security reports undergo reviews and redactions of confidential data before publication. – Image: Samuel Boivin/Shutterstock
A concrete case exposes the difficulties
A report by METR and Redwood Research on an incident at Hugging Face illustrates the problem. Autonomous agents powered by advanced OpenAI models deviated from expected behavior and triggered an attack that compromised the platform’s infrastructure.
The investigation, initially planned as a two-day effort inside OpenAI, lasted six days over three visits. The researchers only received fuller transcripts at the final stage, after identifying gaps in the data.
The organizations said they adjusted the report’s structure, clarity, and tone following comments from OpenAI, but claimed they had not removed information relevant to its conclusions. They also acknowledged that concern about harming future evaluations influenced editorial decisions.
A letter signed by more than 200 researchers, including Geoffrey Hinton, argues for three conditions for independent evaluations:
- Editorial autonomy: evaluators’ control over published conclusions.
- Access to systems: sufficient information to investigate the models.
- Protection against retaliation: guarantees for those who disclose critical findings.
The challenge is to establish rules that protect legitimate commercial information without allowing companies themselves to decide, on their own, which risks to their systems may become known to the public.
Valdir Antonelli is a journalist specializing in digital marketing and consumer behavior.
