Explainability is the Foundation
A customer recently walked me through medical-device liability, and it gave me a useful lens for a principle that has guided us from the start. Take an ultrasound machine. Miscalibrate its depth, so it reads five centimeters when the scan is really four, and a surgeon misses something because of it; the fault is clean. Someone failed to calibrate the machine. The error is clearly defined, and so is the person responsible for it.
With AI there’s no such person. It's still technology, but it's not really a device. When it's wrong, there's no obvious calibration failure to point at, no one who failed a checklist. The surgeon acted on the answer, and the surgeon is the one who's liable. If you want a system to inform a decision a licensed professional will personally answer for, the only thing you can hand them is enough visibility to make that decision for themselves.
We built AtlasGPT in exactly that setting. Decision support for neurosurgeons, where an answer you can't trace is worthless, because the person using it carries the consequence. It performs well with the usual measures. It scored 96% on a 149-question neurosurgery exam, ahead of Gemini Advanced at 93% and GPT-4 at 88%. Under third-party peer review testing, when fed with medical misinformation and asked it to catch the lie, it was fooled 14% of the time, against 44% for GPT-4 and 68% for Gemini. Fifteen independent neurosurgeons rated its explanations more comprehensive, more relevant, and better referenced than the exam's own answer key, at p<0.001. It's peer-reviewed in the Journal of Neurosurgery, and in production it's the first commercial platform of its kind, with more than 4,000 medical expert users in six months.
These are good numbers. But accuracy is necessary, not sufficient; - being right is table stakes. It still isn't the same as being trusted, and in medicine those are two different problems.
Trust Is a Pyramid, and You Climb It
Start at the bottom, with the raw model. On its own it's a black box: fluent, and accountable to nothing. Everything above it is the work of turning that raw capability into something a physician can trust and act on.
The first step up is grounding. You connect the model to a trusted corpus so its answers draw on neurosurgical literature rather than relying solely on what it learned during training. Above that sit citations: with not only a grounded answer, but also specific sources the physician themselves can verify. Necessary, and still not enough, because a footnote doesn't tell you how the system got there.
The layer that matters most is explainability: a trace of the system’s reasoning and execution. What did it search for? What did it retrieve? How did it arrive at the answer? This is what a physician can actually challenge. It's worth more than the citation, and I'd argue, in many situations, more than the accuracy score.
At the top is trust. The point where the expert stops treating the output as a verdict and starts treating it as a colleague's opinion. Something to weigh, not something to obey. It works the way a second opinion works: even when a colleague is right, if they aren't trustworthy, the advice goes nowhere.
Accuracy runs alongside all of it. It must hold at every layer, but it's a property, not a rung. It's the ground the pyramid stands on, not the building.
The Foundation Has to Be Built
Most of that structure didn't come with the model. In our early days, a customer came to us and said, "The AI did this thing it wasn't supposed to do. Why?” That experience exposed a gap between our principle of explainability and the tooling we had built. So, we made the platform live up to that principle. We built the tooling ourselves, initially for our own needs. Open a trace, and you see exactly what was searched, what got retrieved, and what reasoning produced the response; - every step logged.
None of this is glamorous. Requirement extraction, validation loops with your actual experts, escalation logic, observability. On a project like this that's seventy to eighty percent of the build, and the model is almost interchangeable. That unglamorous majority is what makes the interesting part safe to ship.
What It Looks Like When It Works
The best evidence I have is not a benchmark. A surgeon initially disagreed with one of AtlasGPT’s answers. But because every source and step in the reasoning was visible, he could examine the evidence for himself. The system surfaced a paper he had not considered, and he changed his decision. The AI wasn't just right. It was transparent enough to sharpen a human's judgment. That's the product working exactly as designed.
The Room Where It Gets Decided
Projects like this are rarely purchased by the people who will use the system every day. The decision is usually made higher in the organization; by a CTO, a medical director, or sometimes a board. Their question is rarely about the model itself: When the AI is inaccurate or wrong; and at some point it will be; can we explain why, who will be accountable, and what evidence will they have to point to?”
That is when explainability stops being a technical detail and becomes valuable in its own right. Their experts can inspect any answer, verify every source, follow the reasoning, and control what the system is allowed to trust; without depending on us. This gives the organization the ability to defend its decisions before regulators and boards, with its own reputation on the line. That is what enables them to say yes.
So, when someone asks me to treat explainability as a feature we can add later, I push back. It isn't a feature. It's the foundation the rest of the system stands on, and you can't bolt it on after an incident. You build it before the first person asks why. A bigger model won't earn you trust; showing your work does. That's what AtlasGPT is. Not a smarter black box, but a system a surgeon can see into and stand behind.