Why confident wrong answers are the main thing to engineer out of an AI system.

If you are moving a hundred million pounds across a portfolio based on a number, you want to know that number is real. Not approximate. Not a reasonable guess. Real. You want to trace it back to a source. You want to know who measured it, when they measured it, and whether the measurement changed.

This is where every general-purpose AI model gets you into trouble. Not because it is dishonest, but because it is indifferent. It will give you a number with perfect confidence even if that number is completely invented.

The technical term is hallucination. The practical term is a problem that neither you nor the AI can easily spot after the fact.

The confidence problem

A language model does not know what it does not know. It cannot tell the difference between “I have seen this before and I am certain” and “I have never seen this before but I will generate something plausible anyway.” More troubling, both outputs look identical to you. Both come back as calm, complete sentences. Both carry the same tone of authority.

Ask an LLM to calculate 7 × 8 and it will tell you 56 with absolute certainty, because it has been trained on enough arithmetic that the pattern is solid. Ask it to calculate the occupancy rate of a building based on a spreadsheet it has never seen before, and it will also tell you a number with absolute certainty, even though that number might be completely wrong.

“An AI that is always confident is an AI that will eventually fail you.”

Teddy James, Tercero Analytics

The building occupancy example is not hypothetical. It is exactly the kind of task we encounter in every real estate engagement. The spreadsheet has inconsistencies. The column headers change halfway down. The data is in three different formats. The AI will extract a number. It will do so confidently. And if nobody checks that number against the source, that number will become a fact in someone’s decision-making process.

Where hallucination actually costs money

In most use cases, hallucination is an annoyance. You ask for restaurant recommendations and the AI invents a place that does not exist. You ask for a company founder’s name and get it wrong. Frustrating, but low stakes.

The moment your AI system touches a number that moves money, the stakes change entirely. If an AI model confidently states that a property has 92 per cent occupancy when the real figure is 64 per cent, that false confidence feeds into a valuation. The valuation feeds into a loan decision. The loan decision affects capital allocation. Now you are not having an annoyed conversation. You are losing real money based on a fabrication the AI did not know it was making.

This is not a reason to abandon AI for this kind of work. It is a reason to build AI systems that will not let you trust a number that came directly from the model.

How you actually engineer this

The solution is not to use smaller models or to fine-tune harder. The solution is to make confidence conditional. Any number an AI extracts must be automatically validated against its source. If the source does not support the extracted number, the system does not output it. The system flags it for human review instead.

This sounds simple because the architecture is simple. In practice it means building in three parts: the extraction step, the validation step, and the review queue. The AI does the reading. The validation engine does the checking. A human does the thinking on anything that did not pass.

The validation engine is the load-bearing part. It needs to know what was extracted and where it came from. It needs to cross-reference. It needs to be able to say “that number came from column D row 47, and column D row 47 says 64, not 92.” This is machine-readable work. It is boring. It is completely reliable. And it is absolutely non-negotiable if the number is going to move money.

We built this way into every system we run. Every number that surfaces in a dashboard or a report traces back to its source cell. If that source changed, the system knows it changed. If the AI extraction did not match the source, the system caught it automatically before the number ever got to a spreadsheet or a slide deck.

The confidence threshold

Once you have validation in place, a second discipline becomes possible: you can set a confidence floor. If 95 per cent of your extractions pass validation on the first try, then the 5 per cent that fail are immediately suspect. You can flag them. You can weight them differently in reporting. You can require sign-off from a named person rather than letting them flow through automatically.

This is not the AI being intelligent. It is the process being honest. The AI says what it found. The validation step says whether what it found matches reality. The confidence floor says whether reality was clear enough to rely on without additional review.

The firms getting real value from AI in property are the ones that stopped treating the model output as a finished number and started treating it as a hypothesis. A hypothesis you validate. A hypothesis you can overturn. A hypothesis that never touches a real decision until it has been checked against its source.

Your AI should be useful. But first, it should be honest about what it knows and what it is guessing at. That discipline starts by making sure a confident wrong answer is simply not possible.