Building AI Systems That Actually Work

The gap is not the model

Modern models are extraordinary and widely available. Two organisations using the same model can reach entirely different outcomes, which tells you the model is not the variable that matters most.

What separates a system that survives contact with real work from one that does not is engineering: reliable data underneath it, clear boundaries around what it may do, a measurable definition of correct, and a path for a person to intervene. That is the part that takes the time, and the part demos skip.

Why demos succeed and systems fail

A demo runs on chosen inputs, in one session, watched by someone who wants it to work. Production is none of those things.

• Inputs arrive malformed, truncated, in the wrong language, or attached to the wrong record.
• The same question is asked in fifty phrasings, several of which the demo never saw.
• Nobody is watching at 2am, so a confident wrong answer propagates unchallenged.
• The underlying data changes shape, and the system carries on as if it had not.

None of this is an argument against AI. It is an argument for treating an AI feature as a system with failure modes rather than a capability that is either present or absent.

Grounding: answers that can be checked

An ungrounded model guesses fluently. A grounded one retrieves from your own material and can show where an answer came from.

Grounding is less about the retrieval algorithm than about what sits beneath it: sources with clear ownership, permissions that follow the user rather than the index, content chunked so meaning survives, and a freshness policy so yesterday's policy document is not quoted as current.

Evaluation: a measurable definition of correct

Before shipping, we build a test set from real cases and measure against it. Not a public benchmark, those tell you about public benchmarks, but the actual inputs the system will face, with the answers a knowledgeable person would give.

That set does two jobs. It decides whether the system is good enough to ship, and it catches regression later when a prompt, a model version or a data source changes. Without it, 'better' is an opinion.

It also establishes the baseline. If the manual process is 92% accurate and takes four minutes, an AI system at 88% is not an improvement no matter how impressive it looks.

Guardrails: what happens when it is unsure

The important design question is not what the system does when it is right. It is what happens when it is not.

• Confidence thresholds. Below a defined bar, the case routes to a person instead of proceeding.
• Approval gates. Anything consequential (money, external communication, deletion) waits for a human decision.
• Scoped permissions. The system reaches only the tools and records it has been granted, and no others.
• Audit trail. Every automated decision is recorded with its inputs, so it can be reviewed afterwards.
• Graceful failure. An honest 'I do not know' is a correct output. A silent wrong answer is the expensive one.

Integration and monitoring

An AI system that produces good answers into a chat window has not changed the business. Value appears when the output reaches the system that holds the record: the CRM updated, the ticket routed, the invoice reconciled, the draft placed where someone will actually see it.

After launch, quality drifts. Models are updated, business rules change, and the distribution of incoming work shifts. Production monitoring of accuracy, escalation rate and latency is what keeps the system honest over time.

Key takeaways

• Model choice matters less than the engineering around it.
• Ground answers in governed data so they can be checked and cited.
• Measure against a baseline and a test set built from real cases.
• Design the uncertain path first: thresholds, approval, audit, honest failure.
• Integrate with the system of record, then monitor for drift.

Frequently asked questions

How do you know if AI is the right solution?

Measure the baseline first (how the task performs today in time, cost and error rate), then look for work that is high-volume, rules-heavy and expensive in human hours. If a process change or a simple script would do the job better, that is the honest recommendation.

What is grounding in an AI system?

Grounding connects a model to your own governed, permissioned data so answers are retrieved from real sources rather than generated from general training. It makes outputs traceable, because the system can cite where an answer came from.

How long does it take to move an AI feature into production?

It depends far more on the state of the underlying data and the risk of the action than on the model. Work with clean sources and low-consequence actions moves quickly; anything touching money or external communication needs approval design and evaluation first.