Anthropic said it may be close to developing an artificial intelligence capable of accelerating its own improvement. The warning appears in an update to its initial public offering prospectus (S-1/A), filed with the U.S. Securities and Exchange Commission (SEC).
The company says it has not yet reached that stage, but its estimates indicate it could happen soon.
The company already uses artificial intelligence in activities to research and develop new models. – Image: gguy/Shutterstock
What is recursive AI self-improvement?
Known as recursive self-improvement (RSI), the technology describes a stage in which artificial intelligence systems are able to improve their own development processes, accelerating the evolution of their capabilities.
The possibility divides opinions. Critics concerned about the risks of AI say this advance could lead to the creation of systems so powerful that they would escape human control and cause harm while pursuing their goals. Enthusiasts, meanwhile, see RSI as an opportunity to increase productivity and accelerate scientific discoveries.
Anthropic acknowledges both sides of this debate.In the document submitted to the SEC, the company states that it already uses artificial intelligence in its research and development activities, but warns that this strategy could also pose a considerable security risk, especially if it dramatically accelerates the pace of progress.
The company acknowledges that it could reach that stage soon
The update includes new warnings in the risk factors section. Anthropic discloses that it has already used AI systems to develop new models without realizing, at the time, the magnitude of the advances in capability achieved.
The increasing use of AI in research could also cause models to evolve more rapidly than the company’s ability to develop, validate, and implement safety, alignment, and protection measures.
According to the document, internal estimates and projections for future models indicate that the company may soon reach what its Responsible Scaling Policy defines as “dramatically accelerated progress attributable to AI.”
The company notes that none of its current models has been classified as having crossed that threshold. The evaluation of the data remains ongoing.
The company is considering updating its Responsible Scaling Policy to maintain human control. – Image: tete_escape/ShutterstockHuman oversight should continue
Even in light of this possibility, Anthropic says it does not expect to be able to use Claude to design and build successive generations of advanced models in a fully autonomous manner without significant human oversight or understanding.
The company also considers updating its Responsible Scaling Policy to keep people in control of the development of its most advanced models.
The document also deserves to be considered in the context in which it was presented: filings submitted to the SEC fulfill disclosure obligations, but can serve as instruments for presenting the company’s outlook to the market.
Anthropic emphasizes that Claude’s capabilities advanced this year, particularly in coding, mathematics, and cybersecurity. Nevertheless, the company does not consider any current model to have reached the stage of recursive self-improvement.
Valdir Antonelli is a journalist specializing in digital marketing and consumer behavior.
