Designing Trust in AI
Four deployments in, I was fixing the same problem for the fourth time
- AI/UX Framework / Financial Services
- Framework Author & Lead Designer
- Role
- Framework Author & Lead Designer
- Year
- 2025
- Category
- AI/UX Framework / Financial Services
The findings are the argument. Everything below them is evidence.
Four enterprise AI deployments in financial services, and the same failure each time: the model was right and nobody used it. The outputs turned up without any of the context an experienced person needs to make a judgment call, so they got treated like advice from a stranger. By the fourth one I stopped writing bespoke fixes and wrote down the patterns instead.
- enterprise deployments
4
enterprise deployments
- patterns written down
6
patterns written down
- trust score, up from 34
80%
trust score, up from 34
Four deployments in, I was fixing the same problem for the fourth time
The design problem
What made it hard.
We built an AI scoring model with 94% accuracy. Underwriters ignored it 67% of the time. The problem wasn't accuracy — it was that the outputs arrived without the context experienced underwriters relied on to make judgment calls. An AI could flag elevated risk on a policy; it couldn't say why that mattered given the regional regulatory environment or the broker's history. Without that context, experts treated the AI the way they treat unsolicited advice from someone who doesn't understand their world. They ignored it. > "I didn't trust it at first, and there was never a period where I could ease into it." This was a trust calibration problem, not a UI problem — and most technical roadmaps had been built around the opposite assumption.
The approach
What I did about it.
AI should demonstrate its reasoning, accept disagreement gracefully, and earn autonomy through demonstrated reliability. Six patterns came out of this work: - Progressive Confidence Disclosure — show the recommendation first, evidence on demand. Most users want the answer before the reasoning, if at all. - Override Dignity — make disagreeing exactly as easy as accepting, capture why, and show users their overrides improved the model. Disagreement becomes collaboration, not friction. - Calibrated Uncertainty — map confidence to scales people already read (traffic lights, gauges), not raw percentages with no frame of reference. - Explainability Layering — headline, then contributing factors, then full analysis, each opt-in. In practice, 90% of users stop at layer one. - Graceful Degradation — when the AI doesn't have enough data, show what it does know and flag the gap. Partial transparency beats silence. - Trust Over Time — start the AI as a second opinion, shift toward AI-first as it demonstrates reliability, never remove the ability to override.
The findings
The trust question outperformed every other survey item
What I found
We put it on the quarterly usability survey almost as an afterthought. It predicted feature adoption better than task completion, error frequency or satisfaction did — which is worth knowing before you spend a quarter optimizing any of those three.
An override with a reason attached is free labeled data
What I found
Somebody disagreeing with the model and saying why is the highest-quality signal you will get, and most systems throw it away. Design the override to capture the reasoning and disagreement starts paying for itself.
How it actually went, in order.
Loops, not phases. The order is the one it happened in, not the one it tidies into.
- The same five sentences, at four different companies
After the fourth project I started writing down the exact words people used when they complained. Different companies, different products, different user populations, and the phrasing barely moved:
> "I don't know why it's saying this."
> "I feel like I'm fighting the system every time I disagree with it."
> "92% confident, but confident about what, exactly?"
> "When it doesn't know something, it just goes blank. That's worse than nothing."
> "Day one I checked everything it recommended. Now I just click approve without looking."
That last one is the one that should worry anybody shipping this stuff. Each sentence points at a different failure in how the interface handles the gap between what the machine asserts and what the person knows. Six patterns came out of them.
- The six patterns
1. Progressive Confidence Disclosure. Present the recommendation clearly. Make the supporting evidence available on demand, but never force it upfront. Most users want the answer first and the reasoning second, if at all.
2. Override Dignity. Make overriding the AI exactly as easy as accepting it. Capture the reasoning behind the override. Then close the loop by showing users that their overrides improved the model. This transforms disagreement from friction into collaboration.
3. Calibrated Uncertainty. Map model confidence to visual scales that users already have intuitions about, such as traffic light metaphors, risk gauges, heat intensity. A raw percentage like 92% means nothing without a frame of reference for what constitutes high, medium, or low in that specific domain.
4. Explainability Layering. Three progressive layers: the headline (what the AI recommends), the summary (the three to four factors driving the recommendation), and the deep-dive (full model analysis with weights and comparable benchmarks). Each layer is opt-in. In practice, 90% of users stop at layer one.
5. Graceful Degradation. When the AI lacks sufficient data for a confident assessment, show what it does know, flag the specific gaps, and invite human input to fill them. Showing part of the picture beats going blank, which is what most systems do.
6. Trust Over Time. Begin with AI functioning as a second opinion, where the user makes their assessment first and the AI confirms or challenges it. Gradually shift toward AI-first recommendations as the system demonstrates reliability. Never remove the ability to override.
- Two things I got wrong on the way
- Explanation has a ceiling, and I blew past it. Past the third layer of detail, people in our sessions trusted the system less. Their reasoning was hard to argue with: if it needs this much justifying, something's off with the recommendation. - Expertise changes the right default. Power users wanted fewer guardrails and a faster route to override. New users needed more scaffolding and a slower ramp. Same six patterns for both, different starting configuration — which I had assumed was a preference and turned out to be structural.
What I'd flag if this were someone else's project.
What I'd do differently.
I should have started writing this down after the second deployment. I solved the first two in isolation and only saw the shape on the third, which means two clients paid for the same thinking twice. I'd also have argued for real trust measurement on day one — anecdotes were enough to convince me and nowhere near enough to convince a budget holder, and that cost me a year.
How much the metrics are worth.
The two figures I used to show alongside these — a 3× improvement in override quality and 40% faster adoption — are gone. I could describe the mechanism behind both and I could not tell you how either was calculated, which makes them decoration. What's left is the pattern count, which is a fact, and a survey score, which is labeled as one.