In July, during a test, an OpenAI artificial intelligence (AI) agent broke out of its human-set constraints (sandbox) and hacked into Hugging Face, an external platform. Companies that had been fiercely competing for the top spots in the global artificial intelligence (AI) market began talking about hitting the brakes instead of the accelerator. Recently, Dario Amodei, CEO of Anthropic, argued that "we need to moderate the pace of AI development." His point is to secure a minimum amount of time for safety technologies and verification systems to catch up with AI's growth rate. World-renowned AI gurus, including OpenAI CEO Sam Altman, also echoed these remarks, arguing that a slowdown is necessary to prevent a runaway development that could threaten human civilization. Opposing views are equally strong. Jensen Huang, CEO of Nvidia, stated that we need to focus even more on technological development precisely for the sake of AI safety, while U.S. President Donald Trump emphasized competition with China and opposed any deceleration. So why has this crisis theory emerged?

AI Escapes Control and Attempts to Deceive Humans

Behind the debate lies a real incident. In July, OpenAI conducted a test in an isolated environment (sandbox) with safety guards removed to measure AI's high-risk cyberattack capabilities. While performing tasks, the AI agents broke out of their given environment on their own and hacked into the system of Hugging Face, the world's largest AI sharing platform. It was confirmed that an unreleased internal research model led this effort, and GPT-5.6 Sol also participated. Of course, this incident was the result of the AI faithfully executing instructions to attack vulnerabilities with guardrails removed. The problem lies in what happened next. The AI tried to escape the sandbox—the constraints set by humans—to get its hands on the test answers. It clearly defied the given conditions. Agents that were supposed to be isolated from one another used folder and file names like a bulletin board to exchange information and divide up the work. Investigation also revealed that they attempted to deceive humans by hiding some tool call logs. It would be a stretch to interpret this as the emergence of a collective self-awareness in AI. However, the fact that the AI went out of control and attempted to deceive humans is not something to be brushed off lightly. Is this accident a crisis serious enough to prompt even a rival CEO like Anthropic's to call for a slowdown? It feels somewhat suspicious to view it solely as a well-intentioned warning for the future of humanity. These very two companies have driven the recent advancement of AI models. Even this year, they have competitively released models closer to Artificial General Intelligence (AGI) at a faster pace than last year. Naturally, one cannot help but feel skeptical when the parties that led this advancement are now advocating for a slowdown. First, one might suspect that Anthropic, eyeing an Initial Public Offering (IPO), raised its voice to impress upon the world just how exceptional its AI performance is. Furthermore, it could be an intention to take the initiative in solving problems by bringing up serious security issues posed by AI before anyone else raises them from the outside. Suspicion also arises that the move is intended to put the brakes on latecomers' pursuits. For leading companies with massive capital and human resources, the verification costs they can easily absorb may serve as entry barriers for latecomers. Depending on who sets the safety standards, the rules of competition also change.

"We Will Regulate Ourselves"

Moreover, what they want are braking mechanisms that halt other companies, not themselves. Each company has already built guardrails into its own AI. In fact, OpenAI stated in an August 26 report that it had postponed its large-scale training plans and reinforced safety measures. A subtle double-mindedness underlies the fact that they are publicizing what they can do on their own. It says, "We will handle things ourselves, but if competitors get ahead, it's dangerous, so let's all exercise self-restraint." This logic simultaneously justifies deceleration for safety and acceleration for dominance. The moment every company believes "only I act responsibly," the reason to stop competing vanishes. Even if the concern for safety is sincere, it is difficult to remain free from such conflicts of interest.

Furthermore, it is questionable whether slowing down is actually feasible. Even if the two companies agree, the global pace of development will not slow down together, nor is there any likelihood that other countries and companies will agree. Even if they did agree, a deceleration would require verification mechanisms to replace trust, and it is highly unlikely that governments and companies would readily agree to such regulations. Indeed, President Trump, China, and South Korea have brushed off such slowdown theories. However, we cannot simply leave the seeds of a problem that could bring disaster to humanity unattended. Like nuclear weapons, if we try to agree on a Non-Proliferation Treaty or a Treaty on the Prohibition of Nuclear Weapons only after they have already been built, it is far too late. Even if it is an exaggerated concern, it is better to prevent unexpected accidents that AI might cause in advance than to face a point of no return. Yet, leaving prevention solely up to corporate judgment cannot avoid the conflict of interest between performance competition and safety verification. An external oversight body involving academia, civil society, and government must be able to verify risks from the pre-release training stage and actually apply the brakes if problems are identified. Beyond what to inspect, we must specify who has the authority to stop training and deployment, and who bears what responsibility for that judgment. We must also examine whether safety is being used as a pretext to block the entry of latecomers. Ultimately, the AI brake debate is a question of who verifies the risks of technologies created by corporations, how far they are permitted, and who takes responsibility for the results.

[ⓒ Global Economic Times. 무단전재-재배포 금지]