The Mathematical Bullshit Machine: Is There a Missing Step in the AI Investment Boom?

There is an uncomfortable question sitting underneath the extraordinary progress in artificial intelligence.

What if AI keeps getting better — much better — but the thing getting better is not quite the thing we ultimately need?

That is not an argument that AI is useless, fraudulent or about to collapse. The evidence points strongly in the opposite direction. Frontier systems can now write software, analyse documents, generate designs, reason through difficult problems, operate tools and complete increasingly long sequences of work.

The Bank of England recently noted that progress in frontier AI has been faster than many experts expected. AI may indeed become one of the most consequential technologies of our time.

But there is another question that receives rather less attention:

Can increasingly capable generative AI reliably turn an established body of knowledge into a new one through controlled, incremental change?

That sounds like a technical distinction. It may turn out to be an economic one.

The mathematical bullshit machine

The deliberately provocative description I have found myself using for generative AI is a "mathematical bullshit machine."

The mathematics, of course, is not bullshit.

Modern generative models are extraordinary mathematical machines. They learn complex statistical structures from enormous quantities of information and use those structures to generate remarkably convincing and increasingly accurate continuations.

The problem begins when we confuse the product of that process with established knowledge.

At its simplest, a generative system is doing something resembling:

What is the most appropriate continuation given everything available to me?

With sufficient scale, training, context and computational power, the answer can become astonishingly good.

But there is a fundamental difference between:

and:

That distinction matters enormously in engineering.

Engineering is not continually starting again

Imagine an engineering organisation that has developed a powertrain over thirty years.

Its accumulated engineering state contains requirements, drawings, simulations, test results, material specifications, tolerances, manufacturing knowledge, failures, fixes, regulatory requirements, supplier constraints and thousands of decisions whose consequences depend upon one another.

An engineer is asked to change one component.

The engineering problem is not:

"Design me a plausible new powertrain containing this change."

It is closer to:

"Make this change, identify everything that necessarily changes because of it, and preserve everything that should not change."

If we call the established knowledge state K(n), the desired transition looks approximately like:

The difficult part is not generating the delta. It is establishing its boundary.

A material substitution may legitimately change thermal characteristics, manufacturing tolerances, fatigue behaviour, maintenance requirements and certification evidence. Those consequences must propagate. But unrelated established knowledge must not silently change simply because the system has regenerated a plausible new representation of the whole problem.

This suggests a requirement that might be called semantic invariance:

Established meaning should remain invariant except where an authorised change, or the demonstrable consequences of that change, requires otherwise.

That is quite different from producing another excellent answer.

Bigger lottery wheel, better lottery wheel

This raises an awkward question about model scaling.

Suppose the next generation of models is ten times better. Then another is ten times better again.

Errors fall dramatically. Reasoning improves. Context windows expand. Tool use becomes exceptional. Models check other models. Agents operate for hours or days.

All of that matters. But does probability eventually become invariance? Not obviously.

A system that has a 99.999% probability of making the correct transformation is an extraordinarily reliable probabilistic system. It has not necessarily become a system that can establish that an unauthorised semantic transformation did not occur. That is a different computational property.

This is why I use the lottery-wheel analogy. It should not be taken literally: AI output is not random in the everyday sense. These are highly conditioned probability machines. Scaling can make the lottery wheel fantastically good. The question is whether making the wheel larger and better eventually transforms it into something other than a wheel.

Humans aren't perfect either

There is an obvious objection. Human engineers make mistakes.

They miss dependencies, misunderstand requirements, introduce regressions, inherit undocumented assumptions and occasionally make catastrophic errors. So requiring perfect semantic invariance from AI before permitting it to perform engineering would establish a standard humans themselves cannot meet.

But that observation leads somewhere interesting. We don't actually trust individual human engineers either.

Engineering organisations surround fallible human intelligence with requirements management, configuration control, drawings, simulations, testing, design reviews, FMEA, standards, certification, version control, independent verification and audit.

Human engineering is already:

When an engineer proposes a modification, the organisation has mechanisms for determining what changed, what else is affected and whether the resulting state remains acceptable. So perhaps AI does not need to become deterministic. Perhaps its proposals need to be settled by something that is.

The problem is that we do not yet appear to possess a general machine equivalent capable of settling arbitrary semantic change. Formal verification can prove specified properties. Tests can establish tested behaviours. Hashes can establish byte integrity. Version control can identify textual or structural differences. Knowledge graphs can represent explicit relationships. Digital twins can model defined systems. AI models can inspect the work of other AI models.

Each solves part of the problem. None obviously establishes, for arbitrary knowledge:

Everything that needed to change has changed, and everything else has retained its established meaning.

That may be an extraordinarily difficult problem. In the completely general case it may even be impossible. But bounded engineering domains could prove much more tractable precisely because engineering has spent centuries turning ambiguous reality into specifications, units, equations, tolerances, interfaces, standards and tests.

That is where this becomes an engineering question rather than a philosophical argument about AGI.

The economic question hiding inside the technical one

Why does any of this matter to investors? Because the amount of money now being committed to AI is extraordinary.

Gartner forecasts worldwide AI spending of approximately $2.6 trillion in 2026, including about $1.43 trillion on AI infrastructure alone. Gartner also notes that organisations continue to face difficulty demonstrating tangible business outcomes from AI investment.

Meanwhile, the Bank of England says the pace of AI infrastructure investment is historically unprecedented. It identifies 2025 as an inflection point at which required investment exceeded the ability of AI-focused companies to finance it from internal cash flows, increasing reliance on external financing. That financing is spreading through public debt, private credit, leveraged finance, structured finance and off-balance-sheet arrangements.

None of that demonstrates a bubble. But it makes the assumptions underlying those investments increasingly important.

One implicit economic progression appears to be:

What if there is a missing arrow? What if:

That would not mean AI had failed. It could mean AI succeeded technologically while disappointing financially.

Generation productivity is not necessarily lifecycle productivity

Suppose AI makes the initial production of engineering work one hundred times faster. That would be extraordinary.

But imagine that maintaining, verifying, integrating and safely modifying that work remains expensive because the system cannot reliably control cumulative state. The productivity improvement might then look very different when measured across the entire engineering lifecycle.

The economically relevant equation isn't simply:

It is closer to:

And even that may miss the deeper issue. The important measurement may be controlled-change productivity: how cheaply can a system make the 100th dependable modification to something it previously created?

That is considerably less glamorous than asking an AI to design something spectacular from scratch. But it is much closer to what engineering organisations actually spend their lives doing.

There is already evidence for caution — but not yet for the conclusion

The economic evidence remains genuinely mixed.

McKinsey's global AI survey found that almost all respondents were using AI somewhere, and individual use cases were reporting cost and revenue benefits. Yet nearly two-thirds of organisations had not begun scaling AI across the enterprise, only 39% reported any enterprise-level EBIT impact, and most of those reporting an impact attributed less than 5% of EBIT to AI.

The Federal Reserve Bank of St. Louis has found something more encouraging. Reported generative-AI time savings amounted to about 1.6% of total working hours in its 2025 data, and its modelling suggested AI might have increased aggregate labour productivity by as much as 1.3% since ChatGPT appeared. The researchers appropriately caution that the relationship is not straightforwardly causal and that time saved does not automatically become productive economic output.

These are not contradictory findings.

AI can be genuinely productive while the ultimate magnitude of its economic return remains uncertain. And that distinction matters when trillions of dollars are being invested in anticipation of what comes next.

What would actually constitute a warning?

It would not be evidence that AI sometimes hallucinates. We already know that.

It would not be one disappointing model release. Nor would it necessarily be slowing benchmark improvements.

The much more significant finding would be evidence of persistent divergence between:

Imagine that model capability continues rising rapidly while the cost of safely incorporating its output into established engineering, scientific, financial and organisational systems begins to plateau. Then increasing computational expenditure could continue producing spectacular demonstrations without producing proportionate lifecycle economic value.

That would matter. A lot. Because ultimately somebody has to turn all this investment into cash.

The possibility we should hope for

There is, however, an entirely different outcome. Engineers may solve this.

The solution may not exist inside the generative model at all.

Future systems could combine probabilistic intelligence with formal representations, deterministic tools, executable requirements, digital twins, proof systems, configuration management, evidence graphs and entirely new technologies for controlling semantic state. The architecture might eventually become something like:

In such a system the AI remains probabilistic. The cumulative knowledge system does not simply accept probability as authority. That could be enormously powerful. Indeed, it might be the missing step between generative AI and genuinely autonomous engineering.

But notice what has happened. We haven't solved the problem merely by building a bigger language model. We have invented another piece of technology.

So, is AI a bubble?

I don't know. And that is precisely why I think this question deserves more attention.

The evidence already demonstrates that AI is useful. It also demonstrates extraordinary improvements in capability. It would be foolish to dismiss either. Equally, the financial system is committing extraordinary quantities of capital on assumptions about future adoption, productivity and economic returns. The Bank of England now explicitly treats those assumptions, and the increasing debt used to finance AI infrastructure, as matters relevant to financial stability.

The question therefore isn't whether AI works. It is whether it ultimately works enough.

More specifically:

Does increasing generative capability produce a corresponding increase in dependable, cumulative, economically valuable work?

If it does, today's extraordinary infrastructure build-out may eventually look prescient. If engineers discover architectures capable of turning probabilistic generation into controlled cumulative intelligence, we may be watching the construction of something much larger than today's AI industry.

But if capability keeps increasing while verified lifecycle productivity plateaus, the financial consequences could look very different. The technology could still change the world. The investment cycle built around it could still have been excessive. History contains plenty of technologies that succeeded after the financial assumptions surrounding their first great investment boom failed.

So perhaps the question investors, engineers and AI researchers should be asking is not:

"How intelligent will the next model be?"

It is:

"Can it change one established thing without unnecessarily changing everything around it — and can we prove that across the thousand changes that follow?"

Until we can answer that, another turn of the mathematical lottery wheel — however extraordinary — should not automatically be confused with cumulative intelligence.