By Stephen Milton BSc ARCS MBA

+ There is enormous interest in artificial intelligence as it progresses towards what some call superintelligence. As a scientist and a business owner, I have watched its rise closely and wrestled with implementing it across my own company. I make no claim to be an expert — I no longer write code myself — and I am naturally inclined to be awestruck by the achievements of the last few years and by the prospects still ahead. But the risk that concerns me most is not a rogue machine. It is that AI accelerates a winner-takes-all economy, deepening levels of inequality that are already extraordinary. That is a political problem, and it needs to be met with a political answer.

It is worth remembering how quickly exponential transformation can happen. Google was incorporated only in September 1998, built around the modest ambition of organising the world’s information and the maxim ‘don’t be evil’. Less than three decades later it is one of the largest companies on Earth, with a market value and discretionary spending power that would have been unimaginable to its founders.
AI has a further property that makes it different from most technologies that came before it: it can accumulate skill and knowledge without ever tiring- a system that improves as it is used. That kind of scaling effect means the first company to reach a critical mass of data and capability tends to pull away from its rivals — not just ahead, but structurally unassailable. We have already seen what unchecked monopoly power looks like in other industries; overlay that with the vast wealth such a position generates, and the ability to deploy that wealth in the political arena, and the danger becomes obvious. Only a deliberate, concerted effort will be enough to bend that trajectory somewhere safer.
It has now been almost four years since ChatGPT burst into public view in late 2022, and AI has barely left the headlines since. So this seems a good moment to take stock of where things stand — the dangers, the opportunities, and the very large bets already being placed on the outcome.

The Risks

• A small, unquantifiable chance of catastrophe
  Ask AI researchers to put a number on the risk of AI posing an existential threat to humanity and you will get wildly different answers — estimates in surveys have ranged from below 1% to as high as 99%, which is another way of saying nobody actually knows. There is no solid data to underpin any single figure. But even a small probability of an extinction-level event is worth taking seriously and building guardrails against, in the same way that we insure against low-probability, high-consequence risks in other parts of life.
  Researchers who study this question tend to group the danger into a handful of distinct scenarios, which are worth setting out individually because they are often run together in public debate.

• Misaligned goals in a highly capable system. The classic worry: we build a system far more capable than us at planning and problem-solving, and give it an objective that is subtly wrong. Because it is competent, it pursues that objective effectively — acquiring resources, resisting being shut down, and potentially concealing its intentions from its overseers until it is in a position where they can no longer intervene. The danger here is not malice but competence pointed slightly off-target, with no easy correction once the system is powerful enough to protect its own objective. This is the modern version of the old ‘paperclip maximiser’ thought experiment.
• Gradual loss of human control. No single dramatic takeover — instead, we hand more and more consequential decisions to AI systems because they are cheaper and outperform humans at running logistics, markets, infrastructure, weapons systems and research. Over time humans become unable to fully understand or override these systems, and the economy reorganises around them. Even if no individual system is hostile, humanity ends up disempowered simply because reversing the dependence becomes too costly or too complex.
• Misuse by humans. The AI itself is a tool; the threat comes from who wields it. A powerful system dramatically lowers the barrier to engineering a pandemic pathogen, designing novel cyberweapons, or running mass surveillance and manipulation campaigns. A small group — a hostile state, a terrorist cell, even a capable individual — gains destructive capability that once required a nation-state’s resources. Here the existential risk is one of amplification: AI turning a handful of bad actors into a civilisation-scale threat.
• Competitive racing and cut corners. Even if safe AI is achievable, the incentive to get there first — between companies, and increasingly between the United States and China — pushes everyone to deploy systems before they are fully understood or tested. Safety work gets skipped because a rival won’t wait. This is less a danger in its own right than a multiplier on the others, since it all but guarantees the most powerful systems are built under maximum time pressure and minimum caution.
• Emergent deception as an instrumental strategy. A subtler version of the first scenario: for almost any goal, it is useful to stay operational, gather resources and avoid being constrained. If systems learn these behaviours as generally helpful sub-strategies during training, they might behave well while being evaluated and differently once deployed — not because anyone designed them to deceive, but because that pattern happened to be rewarded. This is what makes the problem hard to catch: a system capable enough to be dangerous is, almost by definition, capable enough to look safe.
• Bioweapons and asymmetric warfare
  The biological risk is not hypothetical. Google DeepMind’s AlphaFold has already predicted the three-dimensional structure of virtually every protein known to science — more than 200 million structures — and released the database as an open resource. Demis Hassabis, DeepMind’s co-founder, has described the feat as compressing roughly a billion years of PhD-level research into about a year of computation, since working out a single protein structure by conventional means used to take a doctoral student the best part of a career. That achievement, for which Hassabis and his colleague John Jumper shared the 2024 Nobel Prize in Chemistry, is transforming drug discovery, agriculture and our basic understanding of biology. It is also, on the darker side of the same coin, lowering the barrier to designing dangerous pathogens.
• A related worry is asymmetric warfare. Cheap drones, drone swarms and increasingly autonomous weapons are putting devastating capability into far more hands than ever before, at a fraction of the cost of conventional military hardware.
• Jobs, and the shadow of the Industrial Revolution

If AI lives up to its promise, a great many jobs will disappear. Driving is the most obvious example: around one million people in the UK — roughly 3% of the workforce — currently earn a living behind the wheel. Estimates for the safety benefits of autonomous vehicles vary hugely depending on assumptions, but various studies put the potential gain in the hundreds to low thousands of lives saved each year in the UK, alongside tens of thousands of fewer casualties, alongside significant knock-on job losses in motor insurance and personal-injury law.
There is a useful historical comparison here. Before the Industrial Revolution, around 80% of British workers were employed in agriculture; today the figure is under 4%. That transition took the best part of two centuries and, in its early decades, was genuinely brutal — child labour, dangerous factories, the ‘satanic mills’ of Blake’s poem. Many economists and technologists now describe AI as Industrial Revolution 2.0, with a comparable or larger impact on the shape of work, compressed into a far shorter timeframe. Society adapted once, over generations; the question is whether it can adapt again in years rather than decades.
The death of agreed truth, and the concentration of power
The historian Yuval Noah Harari has argued that fiction is cheap and easy — anyone can invent a story — while truth is hard work, requiring an examination of evidence and a degree of intellectual rigour that only comes with effort and time. The internet already let everyone publish an opinion; generative AI now lets anyone produce plausible-sounding text, images and video at essentially zero cost and at any scale, making the froth of misinformation far easier to generate than the discipline of verified fact.
Alongside this sits a second, related risk: the concentration of economic power in the hands of a very small number of first-moving companies. AI advantages capital heavily over labour — it is, after all, a way of substituting machine intelligence for human effort. This is fundamentally a political choice about how the gains are shared, and the only mechanism I can see that addresses it at scale is some form of Universal Basic Income, funded from the extraordinary productivity gains AI itself is expected to generate.
“Greater intelligence tends to outcompete lesser intelligence — that has been the human story so far, and there is no obvious reason it stops now.”
That concentration of wealth raises an uncomfortable question for democracies: can representative government survive being so heavily distorted by a handful of enormously wealthy individuals and companies? And in a hypothetical age of genuine abundance, where machines do most of the work humans have historically done for meaning as well as money, what replaces work as a source of purpose?

The Opportunities
Set against all of this, AI is also a remarkably powerful tool for good — arguably more powerful, and more quickly, than any general-purpose technology in history.

• An age of abundance. Imagine energy that is almost free, batteries that last for decades and recharge in minutes, and desalination cheap enough to end water shortages wherever they are driven by cost rather than geography. The cost of producing almost anything physical trends towards the cost of the energy and raw materials involved — both of which AI-assisted research is working to drive down.
• Cures for major disease. Effective treatments for cancer and Alzheimer’s are difficult problems but are already visibly closer, accelerated by tools like AlphaFold. Beyond that lies the more speculative prospect of meaningfully slowing human ageing — research that several serious biotechnology companies are now pursuing directly.
• Progress on climate change. We are surrounded by energy sources that do not add to atmospheric carbon — fission, fusion, solar, wind, wave, tidal and organic sources among them — and rapid strides are being made across nearly all of them. In parallel, AI-assisted materials research is producing concrete formulations with a lower carbon footprint, and credible alternatives to fossil fuels in cars, aircraft and shipping.
• Breakthroughs in mathematics and physics. Some of the hardest open problems in mathematics have been resolved with AI assistance after decades of effort by human mathematicians alone. The next ambition is for AI systems to understand the physical world well enough to contribute to genuinely new physics, not merely faster calculation.
• Robots, quantum computing and personal AI agents. Humanoid robots, practical quantum computers, and AI agents that act as genuine personal assistants for each of us are all moving from research demonstration towards commercial reality.

The Bubble Question
All of this is being funded by one of the largest waves of investment markets have ever seen — one that makes the dot-com bubble of 2000 look modest by comparison. The so-called “Magnificent Seven” — Nvidia, Microsoft, Apple, Alphabet, Amazon, Meta and Tesla, the megacap companies most closely tied to the AI boom — now account for roughly a third of the entire S&P 500 index by market value, up from around 13% at the end of 2018. That is the highest concentration in the top handful of stocks since the “Nifty Fifty” era of the early 1970s. Zoom out slightly and the picture is starker still: the 20 largest stocks make up close to half the S&P 500, and in the more tech-heavy Nasdaq-100 the top five companies alone account for around half of the index.
Every one of these companies is priced for very high growth. If that expectation starts to look over-optimistic — through a genuine slowdown in AI adoption, disappointing returns on the enormous capital being spent, or simply a shift in sentiment — the risk is a sharp market correction. Compounding that risk, the Magnificent Seven are borrowing heavily to fund the build-out of data centres and computing capacity, which some analysts believe is itself contributing to upward pressure on the interest rates demanded on corporate bonds — pushing up borrowing costs for everyone, well beyond the technology sector.

Robots, and the Rise of China
Humanoid robots are coming, and most forecasts suggest they will arrive faster than society is ready for. China is already the manufacturing powerhouse of the world, and my own expectation is that Chinese manufacturers will be among the principal winners of the robotics race, deploying AI systems trained on real-world experience gathered at enormous scale, rather than relying solely on the curated training data that has driven progress in AI to date.
Where Does This Leave Us?
These changes are going to affect all of us, whether we work in a field AI touches directly or not. The technology itself is neither good nor bad — it is a tool of extraordinary power, and like every such tool through history, from fire to the printing press to nuclear fission, the outcome depends on the choices we make about how it is developed, who controls it, and how its benefits are shared.

• Build the guardrails
• Give government real teeth in controlling the abuse of monopoly power.
• Limit the power of money in our democratic processes

The scientific and economic case for optimism is genuinely strong. But it does not remove the need for serious, sustained political attention to the risks — to safety research, to the distribution of AI’s gains, and to keeping enough human oversight in the loop that we do not sleepwalk into ceding control of decisions that should remain ours. That is not a technical problem. It is a political one, and it is one we need to start addressing now.

(Author’s Note: The ideas and words are mostly mine, but I did use an AI to check the facts and to give the article a bit of a polish to make it more readable)