How we learned to measure a release
What changed inside Anvia when evaluation became part of how we decide whether a release is ready to ship.
By Anvia TeamFor a long time, a release was easy to describe and difficult to judge. We could point to new capabilities, passing tests, and cleaner APIs. But agent behavior is probabilistic. A build can be correct and still produce worse answers than the one before it.
That gap changed how we work. Across Anvia 0.23 through 0.25, evaluation moved from a collection of useful utilities into part of our release discipline.
The question behind every release
The question was simple: how do we know this version is better?
Traditional tests still matter. They tell us whether an event was emitted, a tool result was preserved, or a memory adapter committed atomically. They cannot tell us whether an answer stayed relevant, whether a summary lost an important fact, or whether a model became more likely to invent evidence.
We needed both kinds of evidence in the same release conversation.
First, build a shared language
Version 0.23 gave us provider-neutral measures for answer relevancy, prompt alignment, JSON correctness, summarization, hallucination, faithfulness, turn relevancy, and knowledge retention. These are not a universal definition of quality. They are a shared language for stating what a feature is expected to do.
We also connected scores to Langfuse experiments and OpenTelemetry traces. When a case fails, we can move from the result back to the generation, inputs, tool activity, and usage that produced it. That changed review discussions from "the score went down" to "this is the behavior that changed."
Then, make a run comparable
Version 0.24 added evaluation run identity and lifecycle reporting. Anvia Lens could finally group a completed suite as one release candidate, compare it with an earlier run, and apply a quality gate to the whole result.
The distinction matters. A metric tells you how one output behaved. A run tells you whether the candidate release is safer to ship.
With stable run identity, we could ask questions that match how we actually ship: Did answer quality regress? Did hallucination improve at the cost of more evaluation tokens? Did a negative control unexpectedly start passing?
Make the result difficult to ignore
Version 0.25 completed the loop in the CLI. Results now include explicit case and metric totals, score direction, target and evaluation usage, optional cost aggregation, abstentions, and negative-control assertions.
Lens can be configured from the environment, bundled eval setup can be reused across suites, and reporters flush at the end of the run. Those details turn an evaluation library into something dependable inside a release command.
The goal is not to claim that one number captures product quality. It is to make every chosen measure repeatable, attributable, and difficult to ignore when a release decision is being made.
What changed inside the company
The biggest change was not an API. Quality became something we discuss before shipping, with evidence that can be revisited afterward. Release notes tell us what changed; evaluation runs help us decide whether that change deserves to reach users.
The first Anvia preview focused on making an agent run. By v0.25, we had a way to compare how it behaved. That is now part of how we want to build the company: move quickly, measure honestly, and make the evidence visible.
Explore every change in the core source changelog, or use the core package documentation for the current public surface.