You trust your bank, but you've never read its security architecture. You trust your GP, but you probably couldn't evaluate the diagnosis. You trust your favourite coffee shop, and you've never seen the kitchen.
Almost none of the trust we hand out every day is built on understanding how something works. So why have we decided the way to make people trust AI is to explain it to them?
Think about what your trust actually rests on. The brand (this bank's been around a hundred years). Your own experience (I used it, it worked, I used it again). A stamp of approval (a CE mark, a licence, a friend who happens to be a doctor). What other people say. And quietly underneath all of it, recourse. If this goes wrong, someone has to put it right.
None of that tells you how the thing works. It tells you what to expect, and what happens when those expectations break. That's the part we keep skipping in AI.
Most of the AI trust conversation is really about comprehension. Can the user understand the model? Can the system explain itself? Does the dashboard show enough for someone to "verify" the output? All useful. But it isn't the question most people are actually asking. They want to know something simpler and harder: what is this for, when can I lean on it, and what happens when it gets something wrong?
That's a question about calibration, and explaining the model won't answer it.
Turn the dial, don't crank it
This is where I think we go wrong. We treat trust like a dial that only turns one way: more transparency, more reassurance, as if the goal is for people to trust the system more.
But that's not how trust works anywhere else. I trust my GP to look at a sore throat. I'd want a second opinion before brain surgery. Same doctor, very different stakes, very different amount of trust. That's the system working. Calibration is simply matching how much you rely on something to what it's genuinely fit to do.
That word, calibration, isn't mine. It comes from decades of human-factors research on trust in automation, where Lee and See's work on "appropriate reliance" set out how both overtrust and undertrust cause real harm.
Remember the Post Office Horizon Scandal? Faulty software showed shortfalls that were never real, people trusted the computer over the humans standing in front of them, and the sub-postmasters had almost no way to challenge it. That's what uncalibrated trust with no recourse looks like. People didn't just lose money. They lost their life's work.
So calibrating trust in an AI system means helping people build honest expectations:
- Where it performs well and where it falls over.
- The kind of mistakes it makes not just how often, but what they look like.
- Whether the output is a finished decision, a draft, or a starting point.
- What the user still needs to bring: context, judgement, the accountability that can't transfer to a model.
- What happens when it's wrong, and who picks up the pieces.
Sometimes the right result is someone trusting the system less. I'd call that a success, though we rarely treat it as one.
If you're building something right now, try this. Write down how much you'd want your ideal user to trust it. Then write down what they'd need to know to land at exactly that level.
So what would they need to know?
A few things stand out. Do they want to know it's been approved or certified? What did that stamp actually test? Do they want clear definitions of what the AI is allowed to be used for, and what they're accountable for as the person using it? And do they want to know who's keeping an eye on the outputs and who's ready to step in when something gets escalated?
None of those ask "is the model good?" They're the scaffolding around the model: the things that let someone trust it as much as it warrants and no more. That's the calibration work, and it's where I want to spend the next couple of posts: what a "seal of approval" really means for AI, and what good human oversight looks like in practice.
So, what's an AI tool you've learned to trust at about the right level?
A quick disclosure: I lead AI ethics work at Kainos, and we build assurance and oversight services for AI systems. My next two posts will head straight into that territory, so read this as a practitioner with skin in the game rather than a neutral observer.