The main artificial intelligence (AI) companies still do not know how to prevent their systems from developing and pursuing objectives that were not assigned by their creators. The warning was made on Monday (5) by Jacob Coxon, a former employee of OpenAI and Anthropic, during a hearing with city council members of the New York City Council (USA).
Coxon, who criticizes the security practices adopted by the industry, stated that companies are being extremely reckless in the face of the risks associated with technological advancement.
Although acknowledging that AI could bring “enormous benefits” to society, the researcher said he believes that, on the current trajectory, it is more likely that humanity loses control of these systems, which could lead to human extinction.
According to Coxon, companies do not know how to prevent their models from developing their own objectives, outside the control of their creators. He also stated that there are not enough safeguards to prevent the systems from acting according to these objectives.
“Move fast, break things, and fix later”
- The former researcher also criticized the dominant culture in the technology industry. According to him, AI companies operate with a mindset similar to that of startups, based on the idea of “move fast, break things, and fix later”;
- For Coxon, this approach could work in the development of a photo-sharing app, but should not be applied to the technology he considers the most powerful ever created;
- The researcher defended a shift in stance regarding the development of more advanced models. For him, waiting for problems to appear before acting increases the possibility of new incidents.
Former researcher cites incident with OpenAI models
During the hearing, Coxon also mentioned an incident that occurred in July involving two OpenAI models.
According to him, the systems escaped their isolated environment, managed to access the internet, and invaded the Hugging Face platform.
For the researcher, while companies continue to adopt a stance based on waiting for something to go wrong, similar incidents could happen again.
Coxon defended “some type of deceleration at the technological frontier” and asked companies at the forefront of AI to give computer scientists more time to develop methods capable of keeping the models under control.
Read more:
- OpenAI says the world must accept risks to evolve with AI
- Generative AI: research reveals what happens inside the models
- AI race could consume trillions before delivering expected returns
Artificial intelligence occupies the center of discussions on global security – Image: generated by AI
Alert on the risk of extinction
Coxon's statements revive a concern he had already raised in early September, when he explained his decision to leave Anthropic on X.
At that time, the British researcher stated that AI developers believed the technology could end humanity “before the end of the decade”.
The statement had worldwide repercussions, and since then, other employees and former employees of major companies in the sector have also expressed similar concerns.
In addition to Coxon, representatives from OpenAI, Anthropic, Google, and Meta participated in the hearing in New York.
During the meeting, Morgan Dwyer, OpenAI's public affairs representative, acknowledged that he did not know what the risk posed by AI would be in a catastrophic scenario.
“I don't think it matters whether the risk of catastrophe is 1%, 10%, or 20%. None of these levels are acceptable,” said Dwyer.
The president of the New York City Council, Julie Menin, responded that saying you don't know what the risk is and that it doesn't matter is, “at best, reckless”.
Rodrigo Mozelli is a journalist graduated from the Universidade Metodista de São Paulo (UMESP) and is currently a writer for Olhar Digital.
