Measuring Expertise
Any large language model can produce a fluent, well-structured answer. The question is whether that answer comes from your expertise or from the AI's general training. We built a way to measure the difference.
The Problem
Modern large language models are trained on vast amounts of text from across the internet. They can discuss virtually any topic with fluency and apparent authority. Ask about M&A valuation or executive coaching or curriculum design, and the response will be articulate, well-organized, and often helpful.
But fluent is not expert. A generalist can describe what a consultant does. An expert knows which framework to reach for in a specific situation, when the textbook answer is wrong, and what question to ask before giving any advice at all. The difference is not in what the AI knows. It is in what it prioritizes, what it leaves out, and how it handles the edges.
This creates a measurement problem. If you build an AI agent trained on your expertise, how do you know whether its responses actually draw on your knowledge, or whether the underlying model is simply producing competent generalist answers dressed up in your voice?
An Expert AI Agent does not replace the language model. It strategically steers it. The model's general intelligence, its ability to reason, structure arguments, and respond naturally in conversation, remains fully intact. What the Expert Agent system does is constrain and direct that intelligence through the lens of a specific expert's knowledge, reasoning, and judgment.
Think of it this way: the language model is a brilliant generalist who can discuss any topic. The Expert Agent system hands that generalist a specific expert's case files, frameworks, communication style, confidence boundaries, and decision logic. The generalist is still doing the thinking. But now it is thinking through the expert's lens, not its own.
This means every response has two contributors: the model's general capability and the expert's configured content. The question is how much of the response's quality comes from each. That ratio is what we measure.
We developed the Expertise Quality Score to answer a simple question: is this response genuinely expert, or is it just generalist AI that sounds professional?
Every response from an Expert AI Agent can be evaluated across four dimensions. Together, they produce an overall score that measures not just whether the answer is good, but whether it is good because of the expert's content rather than in spite of its absence.
Reading the Scores
Each dimension is scored 0 to 100. The overall Expertise Quality Score blends all four, weighted to reflect what matters most. Knowledge Fidelity carries the greatest weight because it measures whether the expert's content is driving the response. Calibration follows, because knowing what you know is fundamental to trust. Voice Consistency and Response Quality share the remaining weight.
A score of 75 or above in any dimension means it is performing well. The expert's content, voice, or boundaries are being used effectively. Between 50 and 74, there is room to improve. The agent may be relying more on the AI's general knowledge than on configured content, or the expert's voice may not be coming through consistently. Below 50, that dimension needs attention. The expert's content may be missing entirely, the agent may not sound like the configured persona, or it may be answering confidently on topics it should defer on.
The scores are designed to be directional, not absolute. A Knowledge Fidelity score of 60 does not mean the response is 60% expert. It means the evaluator found moderate evidence that the response drew on configured content, but also identified claims or reasoning that likely came from the model's general training. That directional signal is what matters: it tells the agent owner where to invest next.
Without measurement, you have no way to know whether your Expert AI Agent is earning its name. A generalist AI that happens to have a professional tone will produce answers that feel good. But those answers draw on the AI's training data, not on your expertise. They will be competent but generic. Correct but not yours.
The Expertise Quality Score makes this visible. When Knowledge Fidelity is high, your agent's responses are grounded in the content you provided: your case studies, your frameworks, your knowledge base. When it is low, the AI is filling in the gaps from its own general knowledge, and the score tells you so.
Most AI evaluation asks: "Was this answer good?" That question has a ceiling. Any modern language model produces good answers. The more important question for an Expert AI Agent is: "Was this answer good because of the expert's contribution?"
That is what the Expertise Quality Score measures. Not just quality. Provenance. Not just whether the answer helps. Whether it helps because of the specific knowledge, reasoning, and judgment of the expert who built the agent.
We believe this distinction is the difference between an AI assistant and a trusted expert advisor. The Expertise Quality Score is how we hold ourselves to that standard.