AI company Anthropic has raised alarms about the risks of self-improving artificial intelligence systems that could outpace human control. In a statement, the firm highlighted that as of May 2026, over 80% of the code merged into its codebase was generated by its AI model, Claude. This level of autonomy signals a turning point in AI development, where systems may soon design their own successors—raising urgent questions about security, monitoring, and alignment with human values.

Anthropic argues that frontier AI labs should establish a coordinated, verifiable mechanism to slow or pause development if systems begin self-improving too rapidly. Such a pause would allow time for societal institutions and safety research to catch up. However, the company warns that unilateral pauses by responsible actors could shift leadership to less cautious developers, potentially increasing global risk.

To address this, Anthropic is calling for agreement among top AI labs on shared triggers and oversight for development slowdowns. Its research arm, the Anthropic Institute, will study technical and governance systems to support such coordination. The company plans to convene discussions with policymakers, civil society, and other AI firms in the coming months. Separately, Anthropic recently completed a fundraising round valuing it at $965 billion and has confidentially filed for a U.S. IPO.