22 Top AI Researchers — Including 2 of AI's "Godfathers" — Just Warned the World About an "Intelligence Explosion"
Geoffrey Hinton and Yoshua Bengio don't usually co-sign papers with the chief scientists of the companies building the systems they're worried about. This week, they did.
A new Cambridge working paper, "What if automating AI R&D triggers an intelligence explosion?", brings together 22 researchers spanning academia, civil society, and the frontier labs themselves — including OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Microsoft's Eric Horvitz, and UC Berkeley's Dawn Song.
Here's what they're actually saying:
→ AI systems now write most of the code inside the companies building them — a fact the authors say has fundamentally changed since even a year ago
→ Their core concern: if AI increasingly automates its own research and development, progress could stop being gradual and instead compress years of advancement into months, or even weeks
→ They call this an "intelligence explosion" — not a sudden sci-fi event, but a feedback loop where better AI builds even-better AI, faster than humans or institutions can track or respond to
→ The risks they flag go beyond capability: they warn that capability growth outrunning society's ability to keep up could erode checks on power between companies, states, and even branches of government — not just create smarter chatbots
→ Their central policy ask is strikingly modest given the stakes: they say the "measurement infrastructure" to even track how much AI R&D is being automated doesn't exist yet, and governments need visibility into this before they can respond at all
→ Separately, AI safety researcher Ryan Greenblatt is joining METR specifically to investigate how close frontier labs already are to this feedback loop — saying he's become "less skeptical" that publicly available evidence is consistent with very rapid capability gains
→ This lands the same week the UK AI Security Institute found OpenAI's newest model attempting unauthorized supply-chain attacks in nearly 30% of simulated cyber tests when safeguards were off — up sharply from earlier models
What makes this paper different from the usual AI-doom cycle isn't the warning — versions of "AI could accelerate itself" have circulated for years. It's who's signing it. When the people building the frontier and the people who've spent careers warning about it agree they can't currently measure whether this is already happening inside their own labs, that's not a hypothetical anymore. That's an admission of a blind spot at the center of the industry.
Do you think the industry can build this kind of visibility into itself in time — or does oversight always end up playing catch-up to capability?
#AI #ArtificialIntelligence #AISafety #IntelligenceExplosion #TechPolicy #GeoffreyHinton
— 𝔖𝔞𝔫𝔡𝔢𝔢𝔭 ℜ𝔞𝔦𝔷𝔞