The same article leads to both conclusions: ship it and stop it

For example, in a weekly design review, we are deciding whether to entrust an AI agent that aggregates sales with everything from changing prices to sending emails to customers.

That morning, the British newspaper The Guardian published AI ‘godfather’ Geoffrey Hinton: it might turn out we were better off without AI, dated November 3, 2026. One person says, “Mr. Hinton himself says that humanity still holds the reins. We can release this feature, too.” Another, looking only at the headline, says, “It means we would have been better off not developing it. We should stop deploying it.”

The same article yields both a decision to release it and a decision to stop it.

Geoffrey Hinton is a researcher who made major contributions to neural network research, which laid the foundations for today's AI, and who shared the 2024 Nobel Prize in Physics with John Hopfield. In this article, the interview phrase "we're still in control" is rendered as “the reins.” It means that humans can decide what AI is used for and which operations it is allowed to perform. The original text does not claim that this state has been proven.

Here is the English passage quoted by The Guardian:

My gut is that we're still in control. We might well be able to do something. It'll only work if we put a big effort into it, but we might well put a big effort into it.

In Japanese, this reads: “My gut is that we're still in control. We might well be able to do something. It'll only work if we put in a big effort, but we might well put in that effort.”

“Gut” is a colloquial term for a person's intuition, rather than the result of research or calculation. “Might” means “may be so.” The condition for it to work is making a big effort. The original text does not assert that the effort will be made either.

If this is shortened to “If we take countermeasures, we can avoid a bad outcome,” three points are lost: that it is an intuition; that it is phrased as “might”; and that a big effort is a condition. Reading only the headline leaves out any talk of a good future from the statement “we might have been better off not developing it.”

In this article, I will frame that enormous effort as a set of items that teams which did not develop the model can check before release.

The process of comparing the pace at which capabilities improve with the pace at which verification catches up is not meant to stop AI from advancing, but to make that advance move at a pace that “safety can keep up with”—that is, to read Dario Amodei’s “We Must Pace the Frontier” from the perspective of an IT practitioner. This article treats his November 3, 2026 remarks as items for teams deploying a system to check.

There are three kinds of statements. We will not turn them into a single conclusion

At the beginning, I quoted only one sentence—an intuition. There are other statements in the interview.

Mr. Hinton says himself that he does not believe a bad outcome is inevitable. He also says that AI might lead to a good future and that perhaps it would have been better not to develop it. Reading only one of the two means ignoring the other.

The next two are predictions about different time horizons. The third intuition is the one described at the beginning.

The outcome within ten years is not a measured probability

Mr. Hinton says that the outcome will become clear within ten years, and perhaps sooner. If the risks can be managed, the result, in his words, will be “radical abundance”: productivity will rise sharply, and many people will become better off. If it cannot, the outlook is a widening gap between rich and poor, major problems surrounding culture, large-scale unemployment, and a highly unstable society. The article does not explain how this projection was calculated, nor is it a measured probability.

The “Chernobyl” in the latter half of 2026 or in 2027 is not a prediction of a nuclear accident

Mr. Hinton says, “There will be a Chernobyl in the latter half of this year or next year. People may really pay attention because of that.” Since the interview was on November 3, 2026, this means the remainder of 2026 or 2027.

According to The Guardian, this is not a prediction of a nuclear accident. Two examples are given: an attack using AI that causes a major drop in power-grid voltage, and the financial system shutting down for a week. He says that society can recover from something on this scale, and that it could prompt national leaders to focus on countermeasures. It differs from the ten-year outlook in both timing and substance, and the two are not treated as the same prediction.

According to The Guardian, Hinton likens the incident in the summer of 2026 to the partial core meltdown at Three Mile Island in 1979. The paper describes the incident as one in which an OpenAI AI agent attempted to intrude on Hugging Face. The metaphor is that Three Mile Island was a warning that came before a larger accident. The intrusion pathway and the containment procedures are framed as operational changes IT engineers should make now, so that the threat of AI agents is not left to either apocalyptic speculation or underestimation. This article distinguishes between the incident that can be confirmed in the report and the metaphor Hinton used to discuss it.

None of the three will be treated as approval for release

So far, we have separated a prediction of a near-term incident, a ten-year outlook, and an intuition that things are still under control. All three are in his own words. They do not rest on the same kind of evidence.

We do not treat a prediction as an incident that has already happened. Nor do we treat an intuition as evidence that safety has been proven. If we limit the agenda to extreme predictions, the current permission design will be deferred. If an intuition alone is the reason for shipping, permission design will, once again, be deferred.

A system that specifies every procedure is different from one that is given only the goal

After sorting out the interview, the next step is to confirm the system being introduced. In conventional business systems, humans define the procedures for processing.

Consider a bank transfer. When a user submits a request, the system checks the balance, determines whether the amount can be transferred, completes authentication, and then sends the money. Developers also write exceptions as conditions. Unexpected failures occur in complex systems. Even so, it is humans who write the rules. What is verified is whether the program operates as designed.

In generative AI agents, only the goal may sometimes be provided.

“Analyze this month’s sales, investigate the problems, and carry out the necessary improvements.”

The agent may construct the workflow itself: retrieving data, choosing an analysis method, connecting to external services, and writing and running a program. The developer does not specify each individual step.

There are more items to verify. It is not enough to check whether it operated as designed. We must also check whether the action selected by the AI is permissible to perform in that business. Against the goal of improving sales, actions such as sending emails to customers without their consent or changing product prices may be consistent with the goal.

Being able to achieve the objective and being permitted to execute the method are separate checks.

Even if a system cannot fully explain its internals, the permitted operations can still be defined

When checking whether a selected action may be executed, a question arises: Is it impossible unless all the model’s internals can be explained?

Enormous numbers of parameters are used to perform calculations inside a large language model. It is not always possible for a person to explain why a particular response or action was chosen. This raises the question of whether people can control a technology whose workings cannot be explained.

Even in a massive cloud, a single engineer does not see the internals of all the software, networks, databases, and external services. Instead, organizations have combined access controls, monitoring, redundancy, failure detection, and recovery procedures to investigate the causes of failures and restore services.

For AI as well, fully explaining internal calculations is not the only thing that needs to be determined in advance. We also need to determine what to permit and prohibit, under what conditions to stop it, and how to restore things when a problem occurs.

Permit reading data for analysis, but not deleting or updating it. Require human approval for transmission to external parties. Prevent transactions above a specified amount from being executed. If an anomalous operation occurs, suspend that agent’s permissions. Natural-language instructions alone are insufficient for these restrictions. Merely writing “This operation is prohibited” does not create a mechanism that rejects the operation at execution time. The application and infrastructure reject it instead.

Restrict the operations that can be executed so that an incorrect judgment does not immediately cause significant harm.

The output is not necessarily the same every time. The system also responds to inputs it did not encounter during training. It may be induced by external text. It may also connect multiple procedures on its own. Existing access controls are insufficient to address these four issues. We continue using those controls as well, while adding countermeasures for these four.

This does not mean reducing the system’s capabilities. The more capable the system, the stricter we make the restrictions on permitted operations.

Designing restrictions on actions outside the model means that security in the agentic AI era is shifting from defense to control—toward designs that control an “AI that acts” without stopping it.

A 99% accuracy rate does not guarantee 100 consecutive successes

Even if you define which operations are permitted, a high accuracy rate alone cannot establish safety. A system that processes information quickly is not necessarily safe. Nor is an AI with a high accuracy rate necessarily able to complete work safely.

Suppose an agent correctly processes 99 out of 100 cases. If the remaining case involves sending confidential information externally or deleting important data, that 99% does not qualify as an operational pass.

When operations are performed consecutively, the probability that all of them will succeed decreases. Assume that each operation succeeds independently with a probability of 99%. The probability that all 100 operations will succeed is 0.99 raised to the 100th power. You can verify this using the following formula in Python 3. It only calculates probabilities and makes no external connections.

The results of this formula are approximately 90.4% for 10 operations, 60.5% for 50, and 36.6% for 100. Real-world failures are not necessarily independent. This figure cannot be used as the availability rate of the system currently in operation. What can be concluded is that, when the number of operations is large, the accuracy rate for each individual operation is not enough on its own.

What needs to be verified is whether the expected results are produced, whether incorrect judgments will not cause major harm, and whether the system can withstand unexpected inputs or attacks. We also examine whether it is possible to trace what was decided and executed from the records, and whether the system can be stopped, isolated, and rolled back when a problem occurs. The same kinds of checks are performed in software quality assurance and security.

In AI operations, in addition to determining whether the correct answer was produced, another check is needed: whether the AI selected an authorized procedure with the proper permissions and operated within the permitted scope. Benchmark scores and the safety evaluations required for operations are assessed separately.

The principle of not relying on the model’s own explanation for the correctness of its output can be found in “Why Leaving Everything to an LLM Is Dangerous—The Limits Revealed by Mathematics and Three Principles for Compensating Through Design.” What this article adds is a note on the precautions required when applying an accuracy rate to consecutive operations.

Even if you stop the process, emails that have already been sent cannot be recalled

After checking accuracy and permissions, a commonly proposed safeguard is that “a human can stop it at any time.” The ability to stop it is necessary. Having a stop button and being able to prevent harm are not the same thing.Having a stop button and being able to prevent harm are not the same thing.

Suppose an agent is using an external service. Even if you stop the execution process, an email that has already been sent cannot be canceled. Processing that began externally may continue, and deleted data may not be restored.

The fail-safe concept in traditional design and defense in depth offer useful reference points. A fail-safe is a design that falls on the safe side when something fails. Defense in depth is a design in which another safeguard prevents harm even if one safeguard fails. If you entrust it with sending large volumes of email, in addition to checking before sending, establish a cap on the number of messages, restrictions on recipients, anomaly detection, and a queue for emails awaiting transmission. Configure credentials so they can be invalidated after a short period and, if necessary, cut off outbound communication entirely. These measures are not substitutes for a stop button; they are safeguards to use together with one.

We do not assume that AI will always make safe judgments. Instead, we assume that it may be wrong and take steps to ensure that damage does not spread when it is. This kind of restriction is close to what is called zero trust in security. Zero trust is not a technology for making AI judgments correct. It is an approach that uses authentication, authorization, and least privilege to narrow the scope affected by AI operations. Least privilege is the principle of granting only the permissions necessary for a task.

As the scope in which AI determines procedures on its own expands, the question also arises of whether it will follow instructions to stop or correct it. In AI safety research, the property of accepting correction or stopping by a human is called Corrigibility (correctability). What Nate Soares and others systematized in 2015 was Corrigibility. At present, there is no broadly agreed method that guarantees every advanced AI will have this property.

Research into the model and system-side restrictions should advance in parallel. We should not rely solely on the model obeying instructions to stop in order to prevent harm.

Even if the problem can be addressed technically, companies and society may not implement the measures

This could be read as saying that appropriate design solves the risk. Hinton’s warning raises another issue: even if safety measures are technically possible, it is not guaranteed that companies and society will implement them.

Releasing a new model quickly makes it easier to attract users. Thorough safety evaluation takes time and money. When competitors are releasing models one after another, prioritizing evaluation delays the release. Companies bear the cost of safeguards. The harm from a major accident can extend to users and society. If one company prioritizes safety while others release first, that company alone ends up at a disadvantage.

In pharmaceuticals, automobiles, aviation, and nuclear power, independent evaluations and regulations have been established to allow both progress and safety verification. Hinton calls for independent evaluation of AI as well, because he believes that verification by someone other than the creators is necessary.

His proposal is not to publish first and then use litigation to correct the problem. Rather, as with the U.S. Food and Drug Administration (FDA), before publication one must submit sufficient statistical evidence to an independent body showing that “the benefits outweigh the harms.” He says that, with thalidomide in the 1950s, publishing first and then suing was not the right course of action. Hinton’s argument is that what large technology companies are saying now should be subject to the same treatment.

The same test used in this analogy cannot be applied to AI as it is.Pharmaceuticals are evaluated statistically for their efficacy and side effects in controlled clinical trials. General-purpose AI has a wide range of uses, and new uses emerge after it is released. Its capabilities also change when it is combined with external tools. Pre-release evaluation alone cannot identify new uses that emerge after release. Post-release monitoring, incident reporting, ongoing validation, and, where necessary, restrictions on use are required.

Not all risks are of the same magnitude. It may not be appropriate to apply the same acceptance criteria to AI that summarizes text and AI involved in controlling critical infrastructure. The level of safety required varies with the application and the magnitude of the harm. Who writes the acceptance criteria is also the question of who writes the rules for AI—these questions need to be considered before “coordination for safety” constrains procurement.

On September 21, 2026, Finnish President Alexander Stubb and Norwegian Prime Minister Jonas Gahr Støre issued A Call for Control of Frontier AI Models. Frontier AI currently refers to models at the highest level of capability. At the heart of the call is a single sentence: AI should be under human command, oversight, and control. It calls on companies to undergo independent evaluations that include mandatory pre-release testing and sufficient access for evaluators to assess risk. It also states that, when tightening AI safety rules, the gap should not widen between countries that can benefit from AI and those that cannot.

The Presidential Office’s The rapid development of AI requires international cooperation says that 34 leaders from 32 countries have participated so far. It is a call by leaders rather than a document adopted as national law. The Guardian reports that Mr Hinton supports the call. The call came first, and the November 3 interview will follow.

For humans to remain in control, technically restricting operations is not enough. Corporate responsibility, third-party evaluation, and societal rules are also necessary. The major effort described by Mr Hinton also includes these three.

The team responsible for implementation first writes down four checks

The release team does not defer its checks until the corporate responsibilities and societal rules from the previous section are in place. There are checks that can be written down first. A division of labor in which only researchers need to address safety does not suit this team. When AI enters a workflow, safety changes not only in the model itself, but also in the design and operation of the systems that invoke it.

Before writing the four, one thing must be decided: whether that task needs to be delegated to AI. What AI can do and what AI should do are not the same. If the conditions can be fully specified, conventional programs may be enough. Choose the technology by comparing the benefits to be gained with the newly emerging risks. Once you have decided to delegate the task to AI, write down the following four.

How much judgment to delegate

Delegate information organization and proposal generation, but leave the final decisions on contracts and payments to humans. Automate routine, low-risk processes, and require approval only for high-risk processes. Write this boundary as a list of callable tools rather than as prompt text.

What to examine in validation

Natural, persuasive prose is not necessarily correct. What the workflow checks is not whether the prose is well written. It is the underlying data and the execution results. If you are running SQL written by AI, in addition to checking that the syntax is valid, check the tables being referenced, the range being updated, and the permissions to execute it.

Where to Contain the Impact When Something Fails

Assume that mistakes will happen. Divide processes into small units, narrow permissions, and retain execution logs. If necessary, stop partway through and make it possible to roll back. If avoiding mistakes is your only goal, the design will leave out how to stop and roll back after a mistake occurs. Make “ensuring that a mistake does not become a major incident” a second goal.

The distinction between reversible and irreversible operations is not about making “human replacement” the goal—it is about applying Mr. Agilere’s Better Path to permission design for AI agents.

Under Whose Name Is It Executed?

The fact that the AI made the judgment does not eliminate operational responsibility. Make who approves, under what criteria it is executed, and who responds when a problem occurs part of the design. A process whose responsible person’s name is left blank remains a verification item to be checked before it advances to automated execution.

Even if writing code takes less time, people determine the scope of delegation

The work of defining the four check items is separate from the work of writing code quickly. What has traditionally been expected of engineers is the ability to deliver the requested functionality. That ability is not becoming unnecessary.

In an environment where AI writes programs, produces tests, and lists candidate causes of failures, it may take less time to build functionality. What is then needed is to verify what to build, how far to automate, and whether the completed system operates safely within business operations.

What you write down is four questions: Is it acceptable to adopt the AI’s proposal as is? Has performance improved without increasing risk? Has human confirmation been skipped for the sake of efficiency? When a problem occurs, who will handle the outcome? Writing these four questions into the permission settings and test items is the work from now on.

What you need to read for this purpose is not limited to how models work. It also includes software engineering, information security, quality assurance, operations, risk management, and the laws and ethics relevant to the scope at hand. You do not need to learn all of them in the same level of detail. Use them to identify which judgments are missing in your own system.

There are areas where AI will shorten working hours. The responsibility for operating technology safely within business and society will not diminish. I wrote about making judgments and taking responsibility alongside people in“Being Able to Use AI” Is Not Enough—The Seven Skills Education Should Cultivate Going Forward, as Gartner’s Forecast Shows. The conclusion is the same. The list of the seven skills is in that article.

Control means that people determine how it is used and which operations are permitted

Having looked at the check items and the engineer’s job, we return to the opening sentence of the interview. “It is still under control” is the interviewee’s intuition. Making it work requires substantial effort.

This sentence does not mean that all risks from advanced AI can be completely controlled with current technology. No such guarantee exists at present. Changes in employment, economic inequality, the spread of false or biased information, and the concentration of decision-making authority in a small number of organizations cannot be solved by access control alone.

Even so, there are things people can decide: which permissions to grant, what purposes to use it for, which systems, having passed which evaluations, to bring into business operations, what restrictions to impose when danger is confirmed, and who should benefit from the gains. These are not questions that will be decided automatically just because performance has improved.

Nor does retaining control mean that human intelligence must remain superior across every challenge.It means maintaining a state in which, even as the number of fields in which AI has greater capabilities grows, people decide what it should be used for and which operations it is permitted to perform. This is a problem of restricting operations through technology, and also a question of which uses society chooses.

In the next review, do not leave any of the four checklist items blank

The meaning of “control” is the same as in the preceding section. It is a state in which people decide how it is to be used and which operations are permitted. What the next design review will write is how to make that state a system configuration.

The review does not end merely with deciding whether to halt or accelerate the progress of AI. It will consider whether evaluation, restrictions, and operations are keeping pace with the rate at which capabilities improve. Performance research and safety research, as well as the detection of new capabilities and unexpected behavior, are all necessary. The implementation team can settle more upstream issues during the review.

How to allocate access permissions. How much of the logging to retain. Which processes should require approval. How to detect anomalies. Where to stop the damage in the event of a failure. These are decisions that have already been made in day-to-day development and operations. The four checklist items are these routine decisions rewritten for agents.

There are both problems that can be addressed with existing technology and problems that require new research, institutions, and cooperation across national borders. There are also decisions to be made by the people who design the system, the organization that decides on its adoption, and those who operate it.

Hinton’s warning need not be treated as a prediction that a bad outcome is already inevitable. It is a call to write down now the specific content of a major effort. Whether it is acceptable to ship will be judged by the items that have been confirmed and written down.

At the next design review, before expanding the tools provided to agents, write down the four checklist items: what judgment to delegate, what to verify, where to stop a failure, and who is accountable for execution. If even one is left blank, that operation will remain subject to confirmation before it can proceed to automatic execution.

Whether humans retain control is not determined by AI performance alone. What judgment to delegate, what to verify, where to stop a failure, and who is accountable for execution. It is also determined by how these four are written into the current design.

Date created: 2026-11-04