Did the version you promoted actually perform better?

Someone changed a prompt three weeks ago. Quality feels different. Nobody can say whether it is better, because the traces from before the change and the traces from after are indistinguishable — neither records which version produced it.

This is the most common gap we see in otherwise well-instrumented agent systems, and it is cheap to close: put the prompt version on the trace, at the moment of the call.

Why the obvious approaches fail

Timestamps. "The change went out around the 14th" works until a rollback, a staged rollout, or a second change in the same week. It also cannot tell you about the run that used a cached prompt from before the deploy.

Git history. Tells you the prompt file changed. Does not tell you which requests saw which version — especially if prompts are loaded from a database or a prompt-management tool rather than the repo.

The prompt text itself. Storing the full rendered prompt on every trace is common and it does work, but it is expensive, it makes grouping awkward (whitespace differences fragment your buckets), and it leaks variable content into what should be a version key.

Store the version identifier, not the version text. One is a join key; the other is a blob you cannot group by.

What we record

Two columns. The discipline is that the value must be set where the prompt is resolved, not where the run starts — otherwise a run that resolves a prompt lazily, or resolves two different prompts in different steps, records the wrong thing.

The comparison

Once the version is on the trace, and scores hang off the same runs, promotion becomes a measurement rather than a feeling.

flowchart LR
    subgraph author["InsightPrompts — authoring"]
        V1["appetite@v1"] --> EXP["Experiment
offline test set"]
        V2["appetite@v2"] --> EXP
        EXP --> PROMO{{"Promote v2"}}
    end

    subgraph prod["Production"]
        PROMO --> RUNS["Live runs
tagged with version"]
    end

    subgraph lense["InsightLense — evidence"]
        RUNS --> SC["Scores per version"]
        RUNS --> CO["Cost per outcome
per version"]
        RUNS --> ESC["Escalation rate
per version"]
        SC --> V{{"Did v2 actually
perform better?"}}
        CO --> V
        ESC --> V
    end

    style V1 fill:#f6f6fb,stroke:#6b7396
    style V2 fill:#efecfe,stroke:#4b34e0
    style PROMO fill:#efecfe,stroke:#4b34e0
    style V fill:#fbede3,stroke:#b4531b

The offline experiment and the production evidence answer genuinely different questions. An experiment runs a fixed test set under controlled conditions — good for ranking candidates, poor at predicting behaviour on the messy real distribution. Production traces have the real distribution and no control. You need both, and the version id is the join.

What we found doing this

A concrete case from our own stack. Two versions of an appetite-assessment prompt, one authored in the actors service and one in our prompt-management product, running in the same period.

The offline experiment said promote v2. The production evidence said v2 is more confident, more expensive, and agrees with underwriters less. That is a promotion you would want to reconsider — and you cannot even have the conversation without the version on the trace.

Where authoring stops and evidence starts

We run two products here and the boundary between them took us a while to get right, so it is worth stating plainly.

The failure mode when this boundary is blurry is duplication: the authoring tool starts storing production traces, the observability tool starts storing prompt text, and you have two partial copies of everything with no agreed source of truth. The version id is the contract; keep everything else on one side or the other.

What this looks like in InsightLense

Because the version identifier is on every trace, InsightLense can put two versions side by side on production traffic — not on a held-out test set, but on the real distribution, measured on outcomes.

The division of labour is deliberate: InsightPrompts owns authoring, versions and experiments; InsightLense owns what happened in production and what it cost. The prompt_version_id is the contract between them, which is why this table can exist without either product storing the other's data.

Doing this in your own stack

• Set the version where the prompt is resolved. Not at run start. A run can use several prompts.
• Use a stable identifier, not a hash of the text. A hash changes when you fix a typo, fragmenting your history for no reason.
• Record the source system too. When prompts come from more than one place — and they will — you need to know which.
• Keep versions immutable. If v2 can be edited in place, every comparison you have ever made is retroactively meaningless.
• Compare on outcomes, not on evals alone. Score, cost per outcome, and whatever your equivalent of escalation rate is. A prompt change that improves one and degrades two is a regression wearing a win.

Two columns on a trace row. It is the cheapest instrumentation on this blog and the one most likely to change a decision.