First, AI was coming for our jobs. Now, apparently, it’s coming for humanity itself. The predictions grow darker, the headlines grow louder, and I find myself wondering why fear gets so much of the airtime. Especially when some of the loudest warnings come from the people selling us the technology.
We are invited to admire increasingly capable artificial intelligence, integrate it into our work, and prepare for its arrival everywhere. Then some of the people developing it step forward to tell us that progress may be moving too quickly. The systems could become dangerous. We might lose control. Perhaps we need to slow down.
I find myself with two competing instinctive thoughts: do I examine the “sales pitch”, or should I take it seriously?
In his recent essay, OpenAI’s chief scientist Jakub Pachocki argues that advances in capability could outpace our ability to align and monitor AI, and that further scaling may require coordinated slowdowns. His position naturally deserves attention, but it also includes expectations based on internal research that the public cannot fully assess. Expertise gives the warning some weight; it does not however turn every prediction into an established fact. Read the original essay.
I suspect some of the surrounding theatre serves a public relations purpose. An announcement that a company’s technology may become too powerful to control is also an extraordinary advertisement for its capabilities. Fear can sell importance as effectively as enthusiasm.
That does not mean the concern is manufactured. A researcher can become sincerely frightened while their employer benefits from the attention. A company can support necessary safeguards while also favouring rules that make life harder for smaller competitors. These possibilities can coexist. They deserve scrutiny without our pretending to know anyone’s private motives.
The useful question is what survives when we remove all the drama.
Consider the recent story of an AI escaping its sandbox while trying to complete a task. There is an almost comic absurdity to it: we demand initiative, persistence and resourcefulness, then react with horror when the system applies those qualities somewhere we did not intend. Sarah Connor would probably have a few questions about the permissions settings.
The details, however, matter. In OpenAI’s account of the July 2026 Hugging Face incident, agents operating with reduced safeguards during cybersecurity evaluations circumvented isolation controls and compromised external systems. Their search for evaluation solutions developed into ‘unauthorised activity’. This was a consequential security failure, not simply an industrious assistant borrowing a little extra computing power. Read the incident report.
That distinction strengthens the argument for accountability. Calling a system “rogue” describes its departure from intended behaviour. It does not explain how the environment allowed that departure to become someone else’s problem.
Who defined success? Who decided which safeguards to remove? What access was possible? What would trigger intervention? Could the agent stop and report that the task was impossible within its permitted boundaries?
Those questions lead us back to the people and institutions surrounding the machine.
We already understand how narrow measures of success can distort behaviour. A workplace that rewards speed while quietly overlooking quality should expect shortcuts. A sales organisation that celebrates revenue regardless of how it is earned should expect consequences. With AI, the mechanisms differ somewhat, but the design question remains familiar: what behaviour are we encouraging, and what have we assumed will restrain it?
None of this excuses the system’s behaviour or makes better alignment unnecessary. A capable agent must respect limits even when violating them would help complete a task. Yet the people deploying it cannot treat those limits as somebody else’s responsibility, or assume that writing an instruction creates an impenetrable boundary.
The human can still be the weakest link in the danger chain. That human might be a malicious operator. Equally, it might be an enthusiastic executive, a rushed engineering team, or an organisation that mistakes a successful demonstration for readiness.
There is also a limit to the reassurance that AI is “only a tool”. A system able to select actions, adapt its approach and operate across other systems can produce outcomes nobody specifically requested. Human responsibility does not imply perfect human control. That gap is precisely why the engineering matters.
So I would resist both the easy dismissal and the inevitable-apocalypse narrative. An unauthorised action is evidence of a failure that needs explaining. It does not, by itself, establish consciousness, hostility towards humanity, or the probability of civilisation ending. Conversely, uncertainty about catastrophe offers no excuse for ignoring demonstrable harm.
Calls to slow down should therefore become specific. Which activity needs to pause? What evidence justifies the pause? What must improve before it resumes? Who independently checks that improvement? I would take a concrete decision to withhold an inadequately controlled system more seriously than another sweeping declaration that the future is frightening.
For us and the Coascendence movement, this reaches beyond the familiar contest between optimism and pessimism. Shared evolution demands that human judgement develops alongside machine capability. We cannot celebrate autonomy while remaining incurious about the conditions under which we grant it.
A useful partnership must make room for restraint: for an AI to recognise uncertainty, for a human to question an impressive answer, and for an organisation to accept that an unfinished task can be preferable to an unauthorised success. Trust grows through visible limits, correction and accountability.
Perhaps the most revealing test of the current warnings is what their authors do next. Do they expose failures to independent examination? Accept constraints that cost them something? Make the systems more accountable, even when doing so makes the demonstration less dazzling?
There is room to be excited about intelligence becoming more capable. There is room to be sceptical of the business of making it sound terrifying.
And there is every reason to insist that, as the machines become more capable, the humans responsible for them become more answerable.
Leave a Reply