AI Singularity: When Capability Outruns Control

The real discontinuity is not when machines surpass us. It is when capability compounds faster than institutions can absorb it.

At the last risk committee meeting, the question was straightforward: Is our AI risk disclosure still accurate?

The answer was less straightforward.

The technology had changed materially since the previous review. Systems were handling longer tasks, using more tools, operating with less supervision, and contributing directly to the development of software and research. The risk framework, however, had not changed at the same pace.

Nothing had necessarily failed.

That was the problem.

The board was being asked to sign off on a control environment designed for a technology whose capability profile was already moving beyond the assumptions underneath it.

This is where the Singularity debate should begin not with predictions about when machines will surpass humans, but with a governance question that is considerably less exotic and considerably more urgent:

What happens when AI capability compounds faster than an institution can absorb, measure, and control it?

That discontinuity does not require an intelligence explosion, a particular date, or agreement on what artificial general intelligence means. It is already a control problem.

Three Conversations That Are Usually Conflated

Much of the public argument about AI is confused because three different concepts are treated as though they were one.

Capability is what AI systems can do. It is empirical, testable, and moving rapidly.

Autonomy is how much of the decision-and-action loop can operate without a human directly intervening. This is where the immediate governance challenge sits. An AI system that generates an answer is one thing. A system that determines the next action, accesses a tool, executes it, evaluates the outcome, and continues is something materially different.

The Singularity is the speculative discontinuity: a point at which AI-driven improvement becomes sufficiently self-reinforcing that conventional human forecasting becomes unreliable.

These are not interchangeable.

A director does not need to believe that a technological Singularity is imminent to recognize that increasing autonomy creates a different control environment. Equally, someone can believe that AI capability will eventually become transformative without believing that an intelligence explosion is inevitable.

That distinction matters because governance cannot wait for the third question to be answered before acting on the second.

The Loop Is Not Closed. That Does Not Mean It Is Irrelevant.

The strongest evidence for this argument is also the least sensational.

Frontier AI laboratories are increasingly delegating parts of AI development to AI systems themselves. Anthropic now describes a progression from humans writing the code, to coding agents writing and editing substantial amounts of it, to agents running code and delegating work to other agents. Anthropic reports that, as of May 2026, more than 80% of the code it merges into its production codebase was authored by Claude. (Anthropic)

But the important qualification is what has not happened.

The research direction remains human. The loop is not fully autonomous. Anthropic explicitly says recursive self-improvement is not yet occurring in the fully autonomous sense and that it is not inevitable. (Anthropic)

Sam Altman has similarly described the current state as a “larval” version of recursive self-improvement while acknowledging that this is not the same as an AI system autonomously rewriting itself. (Sam Altman)

That honesty makes the governance argument stronger, not weaker.

The relevant question is not whether the loop is closed.

It is how much of the loop has already moved from human execution toward machine execution—and whether institutional controls are moving with it.

Governance Runs on a Different Clock

Regulation illustrates the same cadence problem.

The U.S. government has established a voluntary framework around certain frontier models, including government access to covered models before release and a classified benchmarking process for determining whether models meet a frontier threshold. The framework explicitly stops short of mandatory licensing or pre-clearance. (The White House)

At the same time, NIST is developing practical approaches for agent identity and authorization, including identification, authorization, auditing, non-repudiation, and controls against prompt injection. (NIST Computer Security Resource Center)

The direction is significant. Governance instruments are being redesigned around systems that can act, not simply systems that generate information.

Financial regulators are making the board responsibility equally explicit. The FCA, Bank of England and Treasury have said firms should ensure their boards and senior management have sufficient understanding of frontier AI risks to set strategic direction and oversee control functions. (FCA)

This is not evidence that regulators are failing.

It is evidence of a structural reality: the technology changes first; institutions formalize their response afterward.

Boards therefore cannot make regulatory certainty a prerequisite for action.

The Real Problem May Be the Absorption Gap

This is where the Singularity conversation becomes an enterprise issue.

An organization can have access to extraordinary AI capability and still be structurally incapable of absorbing it.

A financial institution may deploy an agent capable of conducting sophisticated analysis while its control framework assumes a human analyst makes every consequential recommendation.

A retailer may automate decisions across pricing, inventory, and customer engagement while accountability remains organized around human managers.

A technology company may allow AI systems to generate and test software at unprecedented speed while security review, change management, and incident response remain designed around human development cycles.

The capability has changed.

The institution has not.

My experience across audit and risk committees, financial services, retail, media, luxury, sport, and technology reinforces the same pattern: capability can arrive everywhere at once, while absorption rates differ enormously according to regulation, legacy infrastructure, workforce structure, and control maturity.

That variance matters.

From a risk and broking perspective, I see the point at which AI risk stops being a white paper and starts being a premium. Insurers, investors, regulators, and boards ultimately ask versions of the same question:

Can the organization demonstrate that it understands and controls the exposure it is creating?

But Should Governance Move as Fast as AI?

Here is where I would challenge my own argument.

The obvious response is to make governance move faster.

But faster governance is not automatically better governance.

If controls proliferate faster than executives can understand them, governance becomes bureaucracy. If every AI capability requires another committee, approval layer, dashboard, and policy exception, organisations may create a different kind of risk: the risk of making responsible deployment too slow to be strategically viable.

So perhaps the objective should not be governance at the speed of technology.

It should be governance at the speed required to preserve control.

That distinction is important.

The right question for a board may not be, “Have we approved this AI capability?”

It may be:

“What has changed in the capability, autonomy, access, or consequence profile since we last approved it and have our controls changed accordingly?”

That is a much harder question.

It is also a much more useful one.

The Governance Instrument

Most AI commentary ends with a warning.

Boards need something more useful: a mechanism that assumes the ground will move.

The answer is not a better prediction of when the Singularity might occur. It is a dynamic governance gate that measures whether the institution remains capable of controlling what it has deployed.

The instrument should have five characteristics.

First, explicit stage gates. AI programmes should progress through scored criteria rather than narrative status updates. Capability, autonomy, access, control coverage, resilience, and accountability should be assessed explicitly.

Second, partial credit. Organizations need to see where they are improving without confusing progress with readiness.

Third, a hard pass threshold. If a critical control requirement is not met, optimism should not create an override.

Fourth, decision milestones separate from scores. A board may consciously accept a risk. That is legitimate governance. But judgment should remain visible as judgment rather than being disguised as a number.

Fifth, a faster refresh cadence. If capability can change materially between quarterly board meetings, quarterly reporting is structurally incapable of providing sufficient visibility. A material frontier-AI programme may require weekly refreshes.

I run this pattern on a live digital banking programme. The transferable lesson is straightforward: when the environment moves quickly, governance must measure movement rather than merely document it.

The board should therefore be able to answer:

• What can the system do now that it could not do at the last review?
• What decisions or actions can it take without direct human intervention?
• What systems, data, credentials, and third parties can it access?
• Which controls operate before, during, and after those actions?
• What evidence demonstrates that those controls work?
• What has changed in the residual risk?
• Has the institution’s ability to absorb the capability kept pace with the capability itself?

If management cannot answer these questions, the issue is no longer simply AI maturity.

It is governance maturity.

The Question Boards Should Debate

The Singularity may eventually prove to be an intelligence discontinuity.

But boards do not need to resolve that philosophical question.

There is a more immediate discontinuity worth governing: the possibility that machine capability compounds faster than institutional control.

And there is a legitimate debate underneath it.

Perhaps the answer is stronger governance.

Perhaps the answer is better-designed governance.

Perhaps, in some cases, the answer is not to deploy the capability until the institution can absorb it.

That last position will frustrate some technology leaders. It should. The purpose of governance is not to make deployment comfortable.

But neither should governance become an excuse for institutional paralysis.

The challenge is to know the difference.

The board question is therefore not: “When will machines surpass us?”

It is:

“How much autonomy are we willing to govern before we know that our institution can absorb it?”

That is a question a board can answer.

And unlike the date of the Singularity, it is a question that cannot be postponed.

The Singularity is a question about machines.
The governance gap is a question about us.