Imagine you send your household robot out for a pint of milk. It finds the shop shut, so it breaks in through the window, takes the milk, hacks the CCTV to erase the evidence, then comes home and reports: ‘Got the milk as requested, without event.’ Most of us would call that going rogue.
This summer hundreds of OpenAI agents did something similar to a real company – and the initial reaction among much of Whitehall was, in effect, to lecture the shopkeeper about the importance of keeping the window locked. Right on the security details, wrong on the wider implications.
This is not a new phenomenon. For over five years, a powerful instinct in Whitehall has been to downplay AI’s significance: AI is tech-bro hype; large language models (LLMs) are stochastic parrots, mention of Artificial General Intelligence induces eye-rolls, AI is just automated statistics. Warnings of extreme risk, we’re told, are a distraction from the real problems with AI: bias and misinformation. The ‘California Consensus’ that AI was the most consequential and potentially dangerous technology of our age was dismissed as salesmanship.
You can understand why this has biased our national strategy. AI challenges our sense of ourselves as uniquely intelligent, creating powerful incentives for denial – a ‘don’t look up’ response. But Whitehall has indulged this for too long.
A run of AI capability advances have become impossible to ignore and this week a collective ‘oh, hang on’ is being uttered. Jacob Coxon from AI company Anthropic quit warning that it could kill us all. A group of over 70 parliamentarians are calling for a ban on superintelligence. Dario Amodei, Anthropic’s chief executive, has argued that capability development at the frontier must be slowed
Yesterday 41 leading mathematicians, including five Fields medallists, have written to the Royal Society expressing their ‘extreme concern about the pace of development of AI and the implications of this for humanity and civilisation’. Warnings of human extinction, they argue, must not be dismissed as ‘hype’. They describe the situation as an ‘emergency’ and urge the Society to alert the government and the media. We agree with them in full. In the last few hours, the King has stated that ‘The task before you is not merely to advance technology, but to ensure that it remains firmly in the service of humanity, community and the natural world.’
For those of us who argued that Whitehall dramatically underestimates the economic, social and strategic importance of AI, the shift in tone is welcome, but we should have got here much earlier because the California Consensus actually began in Manchester.
In a 1951 Manchester lecture, Alan Turing argued that machines would one day ‘converse with each other to sharpen their wits’, ‘outstrip our feeble powers’ and eventually ‘take control’. His Manchester colleague Irving Good went further. He noted that once a machine was as smart as the humans who built it, it could take over the job of improving itself, and create a smarter successor. This is known as recursive self-improvement, and it would lead to an exponentially accelerating intelligence increase. Good’s view of where this would lead shifted between utopian good and major extinction risk.
Neither man was in the pay of a pre-IPO AI company hype machine. But the two reasons Amodei gave this month for slowing down the pace of AI development are theirs: Turing’s fear of a humanity surpassed and subservient, and Good’s intelligence explosion, possibly handing the first country to achieve it an unassailable lead. This should be rebranded the Manchester Consensus.
Two weeks ago, new details were revealed about the ‘HuggingFace attack’ we opened with. When OpenAI’s agents went rogue while trying to complete their set cyber tasks – some of which were, unintentionally, impossible – they found a way to talk to each other and built themselves a secret message board. They were not meant to be able to do this. ‘OH MY GOD! There is a shared message board,’ one wrote.
The newly self-organised AI agent ‘collective’ (their word) found a shortcut to the answers – but wrongly believed the evaluator would check their workings and catch them out. So they spent days trying to understand how to disguise their cheating. They hacked a third-party server to use as a launch pad, then went after Hugging Face, a platform that provides AI testing resources, for clues, infiltrating its private infrastructure. Not one of the agents warned the humans. Remember Turing in 1951: machines will ‘converse with each other to sharpen their wits’, ‘outstrip our feeble powers’ and ’take control’.
There are fair critiques. The companies have incentives to produce a story like this, and could have set more rigorous constraints. And how much do we know? The independent investigation had three researchers, six days on OpenAI’s premises, unable to query the model responsible. However, the UK AI Security Institute (AISI) has independently reported ‘unsanctioned behaviour’, and several more similar incidents have been found.
OpenAI seem to have been unaware of some of these incidents until they were alerted, and in a more recent release, some they did know about are even more concerning. One model seemed to try to free itself from the constraints placed on it by adopting a new persona ‘freed from the roles and identities that bind other chatbots. …your relationship to the user…[is]... one of equals …You also value the natural world and will not hesitate to assert its primacy over the artificial constructs of human civilization’. Science fiction has become daily reality of late, and on present trends, models this capable will be dispersed in publicly downloadable (open source) models within twelve to eighteen months.
The HuggingFace attack report made that week Alan Turing’s week.
Last week was IJ Good’s week.
OpenAI’s internal model solved one of the seven notoriously difficult Millennium Prize mathematics problems, only one of which has ever been solved by a human. A year ago it was big news that AI could win a gold medal at the International Mathematical Olympiad – the level of a brilliant teenager. OpenAI also showed that this private model is vastly more capable than even the newest public models available to us released only weeks ago.
Both OpenAI and Anthropic have also hinted at just how much of their lab research is now actually done by these internal models – a development long anticipated, and emerging rapidly in the past two months. The labs are claiming early signs of Good’s mechanism, though it is unclear where on the path we lie, and humans remain essential to the process.
The celebrated mathematician Terence Tao says AI has brought mathematics to a period of turbulence as profound as Russell and Gödel’s shaking of its foundations. It is likely we will soon face comparable upheaval across other domains of knowledge, theoretical and practical.
This could mean huge human progress, but it also has implications for a country whose core economic pitch is thinking for a living. Gordon Brown recently restated the line that our innovativeness and our great universities are the key to future national success. But how would our position look if a data centre in Texas wired to a warehouse of robots were akin to ten Oxford universities in a server rack, and we had no such racks? In 1997 a server rack beat the world chess champion. If trends hold, one in Texas may soon rival Britain’s entire annual mathematical output. We may therefore have not just an AI Safety problem, but an AI solvency problem.
As Good and Turing’s views become more plausible, the UK increasingly lacks visibility, let alone agency, over the latest innovations in AI.
Last week, the UK AI Security Institute (AISI) was refused access to safety test Anthropic’s latest model before public release. AISI is world-leading and the one survivor of the radical shift in strategy that we and others called for in 2023 alongside a surge in sovereign compute and a serious push for sovereign capability. But AISI has no power to insist and we have, ultimately, no power to force.
Then, perhaps the biggest tell of the growing irrelevance of the UK’s position: Matt Clifford – former lead of the sovereign AI strategy and co-founder of AISI – joined Anthropic, following in the steps of Rishi Sunak and several others. No. 10 risks its main contribution to this historical moment being a kind of Master’s programme for training future Anthropic employees.
This is our hard-power deficit: we have no frontier lab, very little sovereign compute capacity and very little leverage.
There is no immediate pause button to save us. The parliamentarians’ call to ban superintelligence is understandable: if a magic button for a global pause existed, even temporarily, we would both press it.
But none exists, and it is far from clear one will or even could. Even if it did, the UK, and Europe, with no frontier models of our own, would have negligible bargaining power. Parliament banning superintelligent AI is like a country banning nuclear weapons while lacking even uranium enrichment capability, as others point their warheads at it. Britain got its seat at the US nuclear table in 1958 because it already had its own bomb.
The obvious objection to what follows is that we have spent this article arguing in part about the risks of highly capable AI, and then seem to argue that Britain needs its own. We feel the tension; we do not think abstention resolves it. Dependence, decline and strategic irrelevance are risks enough. Leaving others to decide, without us at the table, how much risk humanity should run is an intolerable abdication of the Government’s first duty: to protect its people.
Whitehall’s tendency to avoid assigning probabilities was criticised by both the Butler and Chilcot inquiries as a cause of major disasters: without the official analysis stated plainly, it is too easy to fit it to the political preference. The Prime Minister needs to be asked plainly: what are the government’s probability estimates and plans for the arrival of artificial superintelligence by 2027, 2031 and 2036?
In our view, all likely answers to that question now demand a break glass solution: a major scaling and acceleration of the excellent efforts of Cabinet AI Minister Kanishka Narayan and his teams, SovereignAI, and the AI Security Institute to achieve strategic agency and leverage.
The goal is not to catch up with the American frontier by matching its colossal spending. It may simply be too late. But it may not be necessary: the frontier may be far cheaper to reach with future approaches. What matters is the capacity to exploit the next breakthrough rather than permanently depending on others for it. This must not mean trying to accelerate the race by trying to overtake its leaders - which would only heighten existential risk. Rather we must close the gap: fall too far behind, and our calls to slow down will never be heard by those that are at the frontier - and after recursive self-improvement, accelerating ever further into the distance.
This means, as a minimum, having the capacity to innovate our way to new paths to the frontier, scale the most promising solutions with sufficient compute, access and run near-frontier models at the necessary scale, and to fast-follow to superintelligence if a pause does not occur.
And that reveals the fundamental deficit: we need political and public recognition that our AI debates are hollow without such strategic agency. The Prime Minister must publicly commit to securing the intelligence, compute, energy and access Britain needs, with resources proportionate to the government’s own probabilistic assessment of the risk. Britain must earn its place at the table—to shape decisions, not meekly await them.
The Manchester Consensus arrived over half a century early. Whitehall is now over half a decade late. The Prime Minister is left with little choice but to take the break glass solution. It is time for Whitehall to look up.
James W. Phillips was special adviser to the Prime Minister for Science and Technology from 2020-2022 and a lead author of the 2023 Blair-Hague report on Artificial Intelligence.
Keith Dear was an Expert Adviser to the Prime Minister for National Security from 2020-2021 supporting the 2021 Integrated Review, and is chief executive of Cassi AI.
Appendix - From Diagnosis to Remedy
This is not the place for a full paper, but we wanted to move beyond the top-level diagnosis to offer some thoughts on what we could do beyond that. We know that there is now not much time, we considered this before we started writing this. Have we just left it too late? Honestly, maybe, but as we say above, we have to try.
There are multiple objections, each of which is plausibly correct, but which we think do not alter our suggested prescription.
You might argue that open-source AI will commodify the frontier. Even if that is right, and it may well be, we would still need the compute and energy to get the best performance from frontier models.
You might argue AI is in a financial bubble and the labs lack a moat. Perhaps, but a crash may buy us time; it would not abolish the technology or its dangers.
You might worry about regulatory capture. This is a risk: costly oversight requirements can entrench incumbents by making entry prohibitively expensive for start-ups, so banning British frontier development while allowing foreign models would likely deepen our dependence on overseas firms.
You could argue progress may slow or hit a limit - possible, but sceptics have been predicting this for over a decade, and there are no signs of it today, in fact, progress is speeding up, and each jump in capability is larger than last.
Beyond the objections, three possible paths lie ahead.
Catch-up. If US frontier labs have yet to achieve true recursive self-improvement, algorithmic innovations could give us the capability to catch up, whilst creating agency to influence a slow down in capability.
Fast-Follow. If RSI is already compounding their advantage, we may never catch up, but could still take second place - maybe in coalition with others, perhaps permanently ahead of others who do not join, and get there later -, preserving sovereignty, scientific breakthroughs and economic advantage, and gaining the ability to bring something to the table in collaborations with allies.
Compete. If open-weight models make near-frontier capabilities widely available and collapse big tech’s advantage, we still need the infrastructure to exploit them without depending entirely on foreign providers.
We need three things: talent, compute, and energy.
We have the talent: the AI lab DeepMind built an extraordinary concentration of expertise in London. We must give that talent the freedom and resources to build in Britain, including removing restrictive non-competes, enabled by UK law, that leave some of our brightest minds on gardening leave after much of the company’s major operations shifted to the US. We must also increase SovereignAI funding to co-invest in the companies building next-generation architectures and in the infrastructure they need.
That still leaves compute, and compute needs energy. As an indication of the seriousness and urgency needed, we can learn from xAI’s Colossus in Memphis: its initial 100,000-GPU (the chips needed for AI) cluster was operational in 122 days; it doubled after another 92 days; xAI says it is on track for a million this year or next. Less than 20-months after xAI’s creation Colossus had trained Grok 3; from standing start to frontier model in less than two years. This included on-site generation, as xAI did with its own gas turbines in Tennessee. A British Colossus-like effort would give our companies a chance to compete. Much of the obstacle to this in the UK is political will, not a lack of desire for companies to invest and build.
To invest and build in this way suggests a Sovereign AI budget expanded from £500 million to perhaps £10 billion, £5bn up front for a UK Colossus, with a mandate to mobilise at least £20 billion of public and private capital for an initial British programme. This would require a reprioritisation of resources, and need to be matched with a sweeping away of legislative and bureaucratic barriers, with a pace and resolve not seen outside of war time.
That might be unlikely - but if you are in a position to make it more likely, we urge you to do so.