When Semantic Search Can't Find "PL857"
"What were the risks of PL857?" is the kind of question that should be trivial. PL857 is a production license. We built “AI Nina” for the Norwegian Offshore Directorate to answer this sort of thing from their own literature, and the first version failed on it. Not from a gap in the corpus.
The paragraph that answered the question was indexed and sitting in our vector store.
And AI Nina simply couldn't find it.
The problem wasn't missing data, it was the assumption that meaning alone was enough to retrieve it.
Why couldn't the vector store find it?
A standard RAG pipeline embeds every document chunk into a high-dimensional vector space using a sentence transformer. The user's question is embedded into the same space, and retrieval becomes a nearest-neighbor search using cosine similarity. (ie, documents are turned into numbers using a model; the user question is turned into numbers using the same model; the similarity between the two determines the search results returned).
This works surprisingly well when semantics separate relevant from irrelevant.
It breaks down on identifiers.
To an embedding model, "PL857" is almost an arbitrary token with no learned neighborhood, so it lands nowhere useful, and cosine similarity returns noise.
The passage existed but the geometry of the vector space just couldn't point at it.
Named entities don't rank
Production licenses, well identifiers, seismic surveys, equipment IDs, document numbers - technical organizations are full of identifiers that must match exactly, rather than semantically. They are everywhere in technical corpora.
Dense vector search is often the wrong tool when an exact match to a critical technical term is required to answer the question.
Where do you want the wobble?
What we use instead is less fashionable and more reliable. Terms that must appear become hard constraints. A title or abstract hit counts for more than the same word buried in the body. Underneath sits BM25, a decades-old sparse lexical scorer that still earns its place.
It ranks exact-term overlap with frequency weighting, runs off an inverted index, costs a fraction of a dense lookup, and scales efficiently across very large collections with predictable latency and modest infrastructure.
The fuzziness vector search gives you for free does have value, so we reintroduce it on purpose, at inference, by letting a language model rewrite the query before it reaches the index. That puts the wobbly bit somewhere observable and configurable, not baked into an opaque distance metric.
When a domain model doesn't help
The other assumption worth testing early: a domain-tuned embedding model will understand your domain.
We saw the same assumption fail on a very different project. On a neurosurgery system, we expected a medical sentence transformer to outperform a general embedding model because it understood the vocabulary. It did - but that wasn't the problem. Embeddings are trained to capture semantic similarity based on how terms co-occur in text, not the clinical distinctions that drive diagnosis and treatment. The model placed a schwannoma beside other tumors with similar names and similar descriptions, even though a neurosurgeon would classify and treat them very differently. To the surgeon, those results were confidently wrong and harder to trust than a keyword miss, which at least has an obvious explanation. Fine-tuning expands vocabulary coverage, but it does not automatically reorganize the embedding space around the distinctions that domain experts actually care about.
Having said that, many production systems combine sparse and dense retrieval. The important design question isn't which technique wins in general, but which part of the query each technique should own.
Why won’t the big frontier models solve retrieval?
The reasonable objection is that this is temporary: context windows keep growing and retrieval keeps improving inside the frontier models, so the harness is scaffolding the next release removes. I don't think that survives contact with the mechanics. Models often struggle to consistently use information across extremely long contexts, even when that information technically fits. Attention is quadratic in sequence length, so there is real pressure against an enormous window. Even if every document fit into the context window, you'd still have no deterministic guarantee that the model attends to the paragraph you care about. This rules out a class of use cases.
Even with a perfect, infinite context tomorrow, you cannot serialize a business into a prompt. Data lives in silos because organizations are structured that way, across systems, permissions, and languages. Something still has to find the right passage and put it in front of the model, and that selection is the real work in AI. It isn’t a bug the next version would patch.
The gap is the harness, not a bigger model
We don't build domain-specific models, we build domain-specific harnesses around commercial ones. And our experience shows its value. AtlasGPT, our neurosurgery system, scored 96% on a 149-question board exam where GPT-4 scored 88%, peer-reviewed in the Journal of Neurosurgery. The improvement came from what we built around the model: retrieval, grounding, evaluation, and workflow design, not a larger model underneath.
Evaluation needs humans
Retrieval quality can't be settled by intuition, so evaluation matters as much as the pipeline. We turn subject-matter-expert preference into something quantifiable, scoring answers one to five across a few axes, an automated CSAT run against domain experts, with adversarial questions in the set and answers grounded in the corpus. We do not hand that judgment to a language model alone. LLM judges disagree with human experts on this kind of specialized work, and there is mounting evidence they prefer their own style, scoring their own outputs higher for reasons unrelated to correctness. For expert quality, you need experts in the loop.
Where the accuracy hides
If you are building retrieval over a specialized corpus, the decision that matters most happens in the first week, not the last. Profile your queries before committing to an architecture. Work out what fraction turn on exact identifiers versus semantic recall, test whether a domain embedding model earns its cost against plain BM25 on your data, and measure that with your experts instead of assuming it. Much of what made AI Nina accurate is unglamorous retrieval engineering that rarely reaches a keynote.
In enterprise AI, accuracy rarely hides in the model; it hides in the harness around it.