Ex‑Anthropic employee says AI teams are truly scared for humanity's future

Jacob Coxon, who spent several years at the AI‑safety startup Anthropic before departing, told the BBC that many engineers and researchers working on advanced systems are "genuinely frightened" about the trajectory of the technology. He said the prevailing sentiment among those on the front lines is that without a slowdown in development, the probability of an existential threat from artificial intelligence becomes alarmingly high.

Coxon’s comments come after a period of intense competition among leading AI labs to push larger models and faster deployment cycles. He explained that the pressure to outpace rivals often leaves little room for thorough safety testing, and that internal discussions at several firms now include worst‑case scenarios that were once considered speculative. According to him, the combination of rapid scaling and limited oversight creates a “perfect storm” for unintended outcomes.

While Coxon does not claim to have quantified the exact odds, he emphasized that the risk assessment is based on a growing body of research pointing to alignment challenges, emergent behaviours, and the difficulty of controlling systems once they surpass certain capability thresholds. He warned that if the current pace continues unchecked, the chance that AI could cause irreversible harm to humanity could rise from a remote possibility to a credible threat.

The concerns echo a broader chorus within the AI community, where figures from academia, industry, and civil society have recently called for more robust governance frameworks. Previous warnings from high‑profile scientists have highlighted the need for transparent development practices, external audits, and international cooperation to mitigate potential dangers. Coxon’s remarks add a personal dimension, underscoring that the anxiety is not abstract but felt by the people building the technology day‑to‑day.

Policy makers are now faced with the challenge of translating these warnings into actionable measures. Some governments have begun drafting legislation aimed at regulating high‑risk AI applications, while industry groups are exploring voluntary standards for safety testing and documentation. Coxon suggested that a combination of regulatory oversight, shared safety research, and a cultural shift toward caution could help align development speed with societal safeguards.

Looking ahead, Coxon stresses that the window for effective intervention may be narrowing. He calls for a coordinated effort that brings together technical experts, ethicists, and legislators to establish clear boundaries for AI advancement. Until such a framework is in place, he believes the sense of dread among AI staff is likely to persist, serving as a stark reminder that the future of artificial intelligence remains uncertain and fraught with high stakes.