Anthropic CEO: Government should have power to block dangerous AI deployments
Anthropic CEO – Dario Amodei, the CEO of Anthropic, has called for governmental authority to halt AI deployments that do not satisfy rigorous safety benchmarks. In a recent essay published on his personal website, he outlined his vision for a regulatory framework that would grant regulators the ability to intervene in the rollout of advanced AI systems, emphasizing the need for proactive oversight to mitigate risks to society.
A Pro-Safety Approach in Conflict
The essay, released on Wednesday, expands on Anthropic’s long-standing commitment to prioritizing AI safety. This stance has positioned the company at a crossroads with its rivals and the Trump administration, particularly as debates over AI governance have intensified this year. Amodei argued that the current pace of AI development requires stronger measures than what existing voluntary guidelines provide, advocating for a system that mirrors the stringent protocols of regulatory agencies like the Federal Aviation Administration.
“Frontier AI models, like airplanes, should be required to undergo technical testing and auditing,” Amodei stated. “Their release should be blocked or reversed as a threat to public safety if they do not meet high standards of safety.”
Under his proposed model, AI developers would face mandatory evaluations by third-party auditors to assess risks in four key areas: cybersecurity vulnerabilities, potential use in biological weapon creation, loss of control over autonomous systems, and the unintended consequences of automated research and development. Amodei insisted that these checks would act as a safeguard against the deployment of AI technologies that could endanger humanity, not just public safety.
While endorsing President Trump’s latest executive order on AI testing, Amodei highlighted that his company’s position demands more aggressive action. The Trump order, signed earlier this month after months of delays, established a voluntary process for testing large-scale AI models before their release, but it did not impose mandatory requirements. Amodei pointed out that this approach might not be sufficient, especially as AI systems grow more complex and their societal impact becomes more pronounced.
Industry experts have voiced concerns about the potential for overregulation, with some arguing that voluntary guidelines could still lead to restrictive policies. Amodei, however, suggested that the time may come when even stricter measures are necessary, comparing the most advanced AI systems to weaponizable nuclear materials. “When the most powerful AI systems look less like airplanes or automobiles and more like weaponizable nuclear materials—a threat to humanity rather than ‘just’ a threat to public safety,” he wrote, “the government must have the tools to act decisively.”
Clash with the Pentagon
The CEO’s advocacy for AI oversight has not only targeted developers but also sparked friction with the Pentagon. Earlier this year, a disagreement over safety protocols for AI systems used in domestic mass surveillance and autonomous lethal weapons led to a public rift between Anthropic and the defense department. Defense Secretary Pete Hegseth labeled the company a supply chain risk, restricting the use of its Claude AI model within the Pentagon.
Anthropic challenged this designation, arguing that it was overly broad and typically reserved for foreign adversaries. The company also contested the Trump administration’s directive requiring civilian agencies to stop using its products, a move that followed failed negotiations over safety guardrails in February. Anthropic insisted its technology should not be employed in fully autonomous weapons or for the mass surveillance of Americans, while the Pentagon maintained that such applications were permissible as long as they aligned with legal frameworks.
Following the lawsuit, Anthropic released its most advanced cybersecurity AI model, Mythos, which has since stirred significant debate in Washington. The model, along with other AI innovations, drew attention from the White House, which raised alarms about the potential for widespread safety risks. The release of Mythos underscored the urgency of Amodei’s call for regulatory action, as it demonstrated how rapidly AI capabilities can evolve.
Amodei’s push for mandatory third-party audits represents a departure from the industry’s usual reluctance to cede control to government regulators. By proposing that agencies themselves could serve as auditors, he aimed to create a system that balances oversight with innovation. This approach, however, has drawn criticism from some AI firms that fear it could lead to excessive control over technological progress.
Despite these tensions, Anthropic’s proposal highlights a growing consensus that AI safety must be a top priority. The CEO’s emphasis on technical testing and public accountability reflects a broader shift in how companies and policymakers view the risks of unregulated AI. As the debate over AI governance continues, the outcome of Anthropic’s legal challenges and the implementation of its safety framework will likely shape the future of AI regulation in the United States.
Amodei’s vision underscores the need for adaptive policies that can keep pace with AI advancements. While the Trump administration has taken steps toward voluntary testing, Anthropic’s call for mandatory audits signals a more hands-on approach. This could mean increased collaboration between the private sector and government agencies, or it could lead to further friction as stakeholders grapple with the balance between innovation and safety. The release of Mythos has only amplified these discussions, making it clear that the conversation about AI regulation is far from over.
