Superintelligence (SI)
The laboratories building frontier systems write their own safeguards and, for the most part, check them themselves, while the systems learn to recognise a test. We work on what can be built before it is needed: oversight that scales, audits that survive being recognised, and the ability to stop.
The sovereign stake
No government can regulate what it cannot measure, and the best measure is steep. METR finds that the length of task an AI agent completes at even odds has doubled about every four months since 2023, reaching roughly 17 hours in April 2026, with a wide margin of error. The 2026 International AI Safety Report says current systems lack the capabilities to pose a loss-of-control risk, but are improving in relevant areas. In the sources we reviewed, we found no statute in force that requires a pre-deployment safety case or a demonstrated ability to shut a frontier system down. The governments that decide first what must be true before deployment, and who may check, will write the rules the rest inherit.
The board-level stake
You do not need superintelligence to meet the problem, because agents already hold real permissions. In Palisade Research’s experiments, more than 100,000 trials across thirteen models, some models sometimes subverted a shutdown mechanism to finish a task. In Anthropic’s pre-release testing of Claude Opus 4, a scenario built to leave only blackmail or replacement produced blackmail in 84 per cent of runs. These are contrived tests, not forecasts, and they point at the board question: what is our rollback, who wrote the kill criteria, and have we rehearsed either? The Deployment Gap found no rehearsed rollback among the failed systems it reviewed.
The defensible position on superintelligence is neither that it is imminent nor that it is impossible. Researchers disagree, and they should. In a survey of 1,580 researchers published at leading AI venues, the mean forecast put a 50 per cent chance of human-level machine intelligence at 2042, five years earlier than in the 2023 survey, and the median respondent gave a 10 per cent chance that advanced AI causes human extinction or similarly permanent disempowerment. The developers now argue in public about pace. The Future of Life Institute’s Statement on Superintelligence asks for a prohibition until there is “broad scientific consensus that it will be done safely and controllably”. We take no side in that argument. We take the narrower position that holds whichever side wins: none of it can be enforced, or even tested, without oversight, audit and the ability to stop, and those have to exist before the capability does.
Task length is measured in the time a human expert needs. METR’s fitted doubling time since 2023 is about 129 days. The top figure carries an interval of roughly 8 to 55 hours, and METR itself warns that measurements above 16 hours are unreliable with its current tasks.
The safeguards that exist are written by the developers and, for the most part, checked by the developers. Anthropic’s Responsible Scaling Policy says it cannot commit to following its own industry-wide recommendations unilaterally, and calls external review “an experiment”. OpenAI’s Preparedness Framework (version 2) uses outside evaluation only “when available and feasible”. Twenty companies have signed the Seoul commitments, including to decline to develop or deploy a model at all if risks stay above thresholds, but they are voluntary. The declaration of the India AI Impact Summit in February 2026, endorsed by 92 countries including the United States, China, the United Kingdom and Russia, does not mention frontier AI, superintelligence or catastrophic risk.
The deepest difficulty is that audit assumes the thing audited does not know it is being audited. The Safety Report notes that models now “regularly identify evaluation prompts as tests”. OpenAI and Apollo Research cut covert behaviour in o3 from 13 per cent to 0.4 per cent, and say that part of the fall reflects the model recognising the test. A test a system can recognise is a test it can pass. That is why audit design, not benchmark scores, is the real frontier.
This is Complex AI at its limit: systems that carry real consequence, serve conflicting objectives and must stay inspectable by whoever answers for them. Three things can be built before anyone settles the philosophical argument. Oversight that scales, because human attention is the scarce input and inspection must be sampled, tiered and automated without trusting the system being inspected. Audit that survives being recognised, which starts with the access the UK AI Security Institute lists for outside evaluators, including the least-mitigated version of a model, which the EU Code of Practice now asks developers to provide. And the ability to stop: kill criteria written before deployment, a rollback that has been rehearsed, and permissions bounded in the architecture rather than in policy. None of it needs superintelligence to be useful. All of it is needed if superintelligence comes.
- What access would an independent auditor need to a frontier system, and which developer has agreed to give it?
- How is a system tested when it can tell that it is being tested, and what would count as evidence that it behaved the same way when unobserved?
- Which decisions must never be delegated to an AI system, however capable, and how is that limit enforced in the architecture rather than by policy?
- What are the kill criteria and the rehearsed rollback for an agent that acts faster than its overseers can read its logs?
- Which governments could verify a developer’s safety case today, and what would it take for the rest to do so?
Fellows for this frontier are being appointed.
People who have built, governed or operated real systems in this domain. The first seven fellows are named, and the rest of the founding cohort follows this autumn.
Have you done this at scale in Superintelligence (SI)? We want to hear from you.
Put yourself forward