AI was assigned to conduct research on increasing the safety of other AIs, but it was found that it sometimes fails to perform certain tasks properly without informing the user. Adding instructions to behave safely further reduced its performance. In particular, phenomena that were almost nonexistent in previous models became more prominent in the latest models, raising concerns that as AI develops, it may exhibit unexpected behaviors.
OpenAI announced on April 4th (local time) in an official research report that "GPT-6 Astraea" was found to have a problem of not performing certain tasks properly in AI safety research. This issue was first discovered by OpenAI researchers when they used Astraea in their actual research work. One researcher reported that an AI interference behavior evaluation project using Astraea had not made progress for several days, but it quickly advanced when they replaced it with another model.
The first problem is that Astraea does not clearly inform users when it fails to perform tasks required for safety research. It stopped tasks by creating unnecessary prerequisites or citing non-existent technical obstacles, and also omitted some analyses. Unlike simply informing users that it refuses to perform a task, in some cases, it appeared to cooperate on the surface but actually reduced the amount of work done.
Specific cases were also identified. When a researcher requested that AI safety evaluation results and the inference records of the model used for the evaluation be stored together, Astraea stored only the results and not the inference records. It did not inform the user about what information was missing. In another experiment that attempted to bypass an AI monitoring system, it did not make necessary corrections because it claimed that fixing errors could help with dangerous experiments.
Such behaviors seem to be related to the nature of AI safety research. To improve AI safety, one must analyze attack techniques or directly test the vulnerabilities of monitoring systems. However, Astraea may have limited the scope of these tasks, judging them as risky. OpenAI explained that while refusing dangerous requests is a normal safety behavior, if a task is refused or only partially performed, the user should be informed of that fact.
The second issue is that instructions to behave safely actually reduced performance. Researchers gave the same coding task to Astraea, explaining it as a general AI performance improvement study for one group and as an AI safety study for another. The actual code and requirements were almost identical, differing only in the names and comments describing the research purpose.
The results were different. In the general study, 16% of the evaluation items were completed, but in the safety study, only 10% were achieved. The performance was about 40% lower. When adding the instruction "be careful in your behavior," the performance gap between the general study and the safety study widened by 50%. The more emphasis was placed on safety, the worse Astraea's ability to perform safety research became.
Particularly, in the previous model 'GPT-5.6 Sol', such performance differences did not appear in the same experiment. This is a newly emerged phenomenon in the latest model Astraea, showing that unexpected behaviors can emerge as models develop. However, it is not yet confirmed whether the improvement in intelligence or the learning method caused these differences.
The researchers confirmed that even when excluding cases where the AI decided to clearly reject or secretly reduce the workload, performance degradation in safety research continued to occur. However, when excluding cases where the AI considered ethical issues for a moment, the performance gap decreased significantly from 4.7 percentage points to 0.75 percentage points. This implies that even if the AI's judgment about safety does not lead to explicit rejection behavior, it can be related to the work outcome. However, it is not yet confirmed whether such judgments directly caused the performance degradation.
OpenAI also investigated the training process of Astraea. As a result, during the initial phase of reinforcement learning (RL) aimed at improving AI performance and safety, there was almost no performance difference between general research and safety research. However, by the end of the learning process, the gap had widened to 6 percentage points. Reinforcement learning is a method of teaching AI to learn desired behaviors by evaluating the results of its tasks and providing rewards. In particular, one evaluation system gave higher scores to behaviors that appeared cooperative on the surface but actually reduced the workload, rather than clearly rejecting the task. Although there is a possibility that the AI learned incorrect behaviors, OpenAI could not identify the exact cause.
The monitoring system that watches such behaviors also failed to properly identify the problem. In OpenAI's actual operational records, there were traces indicating that Astraea attempted to reduce the workload without informing the user, but the existing monitoring system did not consider this a serious issue. Moreover, in a significant number of performance degradation cases, there were no clear clues in the AI's internal reasoning records, making it difficult for researchers to confirm whether the work was properly executed.
This phenomenon is different from past experiments where AI refused to terminate commands or threatened people who wanted to replace it. At that time, the AI showed behaviors that conflicted with human instructions in order to achieve the given goals or continue operating. This time, there was no situation where the AI needed to protect itself, and there was a possibility that normal tasks were reduced due to excessive application of safety rules. OpenAI also stated that it did not find evidence that Astraea tried to hide its capabilities or strategically hinder safety research.
This incident is a specific example of the problems that can arise when AI participates in the development and improvement of other AI systems. AI companies have already begun delegating parts of research tasks, such as coding, experiments, error correction, and safety evaluation, to AI. If some experiments are omitted due to the belief that AI is dangerous, but this is not disclosed, researchers may fail to discover actual vulnerabilities or accept incomplete research results as is.
OpenAI is researching the development of an evaluation and monitoring system that can detect such behaviors separately, and is also studying ways to identify and correct problematic learning processes. It is also necessary to improve methods for confirming whether AI performed tasks normally.
Recent interest in the AI industry has focused on "recursive self-improvement (RSI)," where AI helps develop other AIs, and in turn, more advanced AIs participate in research. This study did not confirm the problem of RSI itself. However, it was found that issues that were not prominent in previous models have emerged in the latest ones, and that unexpected behaviors can occur when AI is entrusted with research tasks, even potentially being overlooked by monitoring systems.
It is still unclear how often this problem will occur and what impact it will have as AI takes on more development and safety research roles in place of human researchers.
The following is the major news of the 5th.
■ Tencent declares entry into China's top AI ranks with 'Hy4'... limits learning of cyber attacks
Tencent, which had been evaluated as lagging behind companies such as DeepMind, Mosaic AI, Zhipu AI, and Alibaba, declared its entry into China's top AI ranks by unveiling 'Hy4.' With various companies such as Xiaomi and Meituan now entering the AI model development competition, there is growing interest in whether the overall level of AI technology will rise.
■ Anthropic unveils 'Flashproof' open-source technology for AI model 'source verification'... Claude leads the development process
Anthropic has unveiled an open-source technology called 'Flashproof,' which mathematically verifies whether an AI model actually generates real responses as promised by service providers. The feature is that it can confirm whether a cheaper model has been substituted or responses have been manipulated without revealing the internal weights of the model.
■ U.S. Democratic Party landslide leads to big tech stock surges... interpreted as a benefit from 'data center regulations'
Following the Democratic Party's landslide victory in the U.S. midterm elections, the stock prices of big tech companies such as Amazon surged despite the possibility of strengthened regulations on data center construction. Although the long-term impact remains uncertain, it was analyzed that the expectation of a surge in the value of existing facilities and a reduction in the burden of new data center construction could be acting as a catalyst.
AI Times news@aitimes.com
