Dario Amodei's new essay reads like a Cold War playbook for the AI age

Dario Amodei, CEO of Anthropic, warns of an imminent AI threat in his recent essay "Policy on the AI Exponential." Amodei argues that the rapid development of AI poses significant risks to national security, the financial sector, and critical infrastructure. In the essay, Amodei illustrates the speed problem using a subplot from "Lord of the Rings," where two hobbits try to rouse the slow tree creature Treebeard to defend his forest against an approaching army. Amodei likens Treebeard to the slow-moving political system and the hobbits to those sounding the alarm on AI risks. The approaching army represents the threat posed by unregulated AI, which is advancing rapidly due to scaling laws that enable model capabilities to grow exponentially with more compute. Amodei expects that within one to two years, "Powerful AI" could emerge, equivalent to "a country of geniuses in a data center." He cites the experience with "Claude Mythos Preview," which disrupted the global cybersecurity landscape, as evidence that frontier models carry real risks. Amodei now calls for mandatory testing by qualified third parties across four risk areas: cybersecurity, biological weapons, loss of control over AI systems, and automated R&D that could accelerate those risks. The Advanced AI Framework, released alongside Amodei's essay, proposes a detailed regulatory approach. The framework targets developers of the most capable models, applying to those training models with more than 10^25 FLOP who either generate over $500 million in AI revenue or spend more than $1 billion annually on AI research. These developers would be required to publish a safety framework, provide system cards for high-risk models, release a risk report at least every six months, and report serious security incidents to the agency within 15 days. Amodei's call for binding regulation marks a shift from Anthropic's previous advocacy for transparency requirements. The company had backed transparency bills like SB 53 in California, RAISE in New York, and SB 315 in Illinois. However, Amodei argues that the risks posed by AI are now too great to rely solely on transparency, and that mandatory testing and regulation are necessary to mitigate these risks. The implications of Amodei's warnings and the proposed regulatory framework are significant. If implemented, the framework could lead to a more secure and responsible development of AI, but it could also impose significant costs and burdens on developers. The framework's focus on the most capable models could also create a tiered system, where smaller developers are exempt from regulation but still pose risks. Amodei's essay and the Advanced AI Framework have sparked a debate about the need for regulation in the AI industry. Some argue that regulation could stifle innovation, while others believe that it is necessary to prevent catastrophic risks. As the development of AI continues to accelerate, the question of how to balance innovation with safety and responsibility will become increasingly pressing. In conclusion, Amodei's warnings about the AI threat are a call to action for policymakers, developers, and the public. The proposed regulatory framework offers a potential solution, but its implementation will require careful consideration of the trade-offs between security, innovation, and cost. As the AI landscape continues to evolve, it is crucial to prioritize responsible development and mitigate the risks posed by this powerful technology.