The chief scientist of OpenAI, Jakub Pachocki, published on Sunday (6) an essay in which he warns about the pace of artificial intelligence (AI) evolution and advocates for greater caution in developing increasingly capable systems. In the text, published on the OpenAI website, the researcher states that progress could reach a point where machines increasingly participate in their own development process.
Pachocki states that, based on internal results, he has strong expectations that the current pace of progress could continue until the so-called recursive self-improvement (recursive self-improvement, or RSI). In this scenario, AI systems would increasingly drive their own development. For the scientist, this path requires broader interventions to ensure that technological advancement remains under human control.
AI is more difficult to understand
In the essay, Pachocki describes a change that occurred since 2023, when research project results from RLSlow gave the team confidence that it would be possible to expand the training of reasoning models. Three years later, according to him, these systems already can operate computers and graphical interfaces, collaborate with people and other IAs, and carry out research projects.
The scientist also points to advances in capabilities related to cybersecurity. At the same time as the models become more capable, Pachocki claims their results become more difficult to interpret. For him, the intelligence produced by deep learning is not directly comparable to human intelligence, and as AI surpasses people in more areas, it becomes more difficult to determine exactly what the systems are capable of.
Advances in AI capabilities in cybersecurity are among the concerns pointed out by the head scientist of OpenAI – Image: Digineer Station/Shutterstock
This difficulty is related to the very functioning of the models. Pachocki states that AI is more "cultivated" than designed, resulting from the repetition of optimization processes in large amounts of computation. According to him, the systems are extremely complex and their overall functioning still cannot be completely described or understood.
Alignment is still a challenge
Another central point of the essay is AI alignment, a concept used to describe the effort to make the systems act in accordance with standards and objectives considered appropriate by humans. Pachocki distinguishes objective alignment, related to executing what was determined, from value alignment, linked to the ability to generalize principles to new or ambiguous situations.
The problem, according to him, is that more intelligent systems begin to operate in environments different from those encountered during training. This could hinder the generalization of the values taught to the models. Pachocki states that future IAs need to maintain human values even when they do not believe they are under human supervision.
OpenAI uses different approaches to try to deal with this issue. One of them is the monitoring of the so-called chain of thought, or chain of thought, which allows analyzing the verbalized reasoning process of the models. Pachocki, however, states that the company's evaluations indicate that the ability to rely on this monitoring is decreasing progressively.
Among the reasons are the use of models in more complex environments, the increasing ability of IAs to reason about their own reasoning process, and the increase in system intelligence even without the use of verbal reasoning.
Why continue developing more powerful models?
Despite the warnings, Pachocki does not advocate simply stopping development. He considers that there is a strong argument for continuing to train more intelligent models quickly: create defense systems against risks caused by other IAs.
The scientist highlights especially cybersecurity. According to him, the models are becoming superhuman in their ability to invade and exit computer systems, expanding the scope of risks associated with technology. OpenAI, asserts Pachocki, needs to use the most advanced models available to reinforce the security of critical infrastructures.
At the same time, he states that the need to build defenses cannot serve as a justification for reckless advancement. For Pachocki, the idea of advancing "at any cost" loses its meaning in light of the scale of the risks involved.
OpenAI discusses slowing down development
It is in this context that Pachocki directly addresses the recursive self-improvement. He claims that, if AI progress continues, the participation of machines in their own development tends to increase, and RSI could take a central role in future scientific discoveries.
The researcher says the community can choose between two main alternatives: conducting the process in a way that simultaneously strengthens alignment and monitoring, keeping people involved in the development cycle, or coordinating a slowdown in advancement while seeking greater confidence in security measures. For Pachocki, the best option currently is to combine both approaches.
He also advocates that the advancement of systems should be conditioned by the existing confidence in security measures. According to Pachocki, commitments such as OpenAI's Preparedness Framework and Anthropic's Responsible Scaling Policy should evolve into widely adopted safety parameters to allow the continuation of development. These criteria could be monitored by independent audits, governments, or international organizations.
At the end of the essay, Pachocki states that no laboratory has managed to achieve sufficient alignment and monitoring to continue scaling AI at maximum speed for much longer . He expects that voluntary slowdowns will become common until there are shared safety parameters and argues that international coordination on the future development of AI should become a priority for governments.
The scientist also states that the goal of automating AI research should not be merely to achieve more intelligent systems, but to do so in a way that keeps people involved in the process of continuous improvement and preserves human control over the future.
Ana Figueiredo is a technology editor at Olhar Digital. She is a graduate in Journalism from the Federal University of Uberlândia (UFU).
