One score primitive: humans, judges, code checks and users in the same table

Most teams end up with three separate systems of judgement: a table for human feedback, a pipeline for automated evals, and a thumbs-up column somewhere in the product. Because they are three systems, nobody ever compares them — and the comparison is the single most useful thing you can do with any of them.

We use one primitive. A score has one shape, four possible sources, and can attach to a run, a step or a business event.

The shape

{
  "target_type": "run",          # run | step | business_event
  "target_id":   "VS-2027-0832950",
  "name":        "appetite_fit",
  "source":      "judge",         # human | judge | code | feedback
  "value":       0.82,            # numeric | boolean | categorical
  "comment":     "Class code in appetite; TIV above preferred band.",
  "scored_by":   "eval:appetite-v3"
}

Three value shapes, because judgement genuinely comes in three shapes: a number (0.82), a boolean (did it cite a source, yes or no), and a category (correct / partially correct / wrong). Forcing all three into a float is how you end up with a "0.5" that means nothing.

A bug worth knowing about

If you build this, check the boolean case before the numeric case when detecting shape. In Python, isinstance(True, int) is True — so a naive check turns every boolean score into the number 1 and you lose the distinction permanently. We shipped that, found it, and now it is the first line of the shape detector.

Four sources, one table

The sources are not a taxonomy for its own sake. They differ in trust, in volume and in latency, and holding them in one table is what lets you exploit those differences.

Source	Who produces it	Volume	What it is good for
human	A reviewer deliberately assessing an output	Low	Ground truth. The thing everything else is calibrated against.
judge	A model scoring another model's output	High	Coverage. Cheap enough to run on everything.
code	A deterministic check — schema valid, citation present, number in range	High	Facts that do not need judgement. Never disagrees with itself.
feedback	The end user, in the product	Medium	What the person on the receiving end actually thought.

The comparison that pays for the whole design

Here is a real referral from our underwriting product, scored three ways.

flowchart LR
    SUB["Submission
VS-2027-0832950"] --> RUN["Agent run
appetite assessment"]
    RUN --> J["judge
appetite_fit = 0.82"]
    RUN --> C["code
pack_complete = true"]
    RUN --> H["human
underwriter = reject"]
    J --> D{{"Judge said yes,
underwriter said no"}}
    H --> D
    D --> A["Calibration gap:
judge over-scores
TIV above band"]

    style SUB fill:#e2f5f9,stroke:#0e97b0
    style J fill:#efecfe,stroke:#4b34e0
    style C fill:#efecfe,stroke:#4b34e0
    style H fill:#fbede3,stroke:#b4531b
    style D fill:#fbede3,stroke:#b4531b

The judge scored 0.82; the underwriter rejected it. That disagreement is the signal. It is only visible because both scores hang off the same run — in separate systems they are two facts nobody ever puts side by side.

One disagreement is noise. Forty disagreements with the same shape is a finding: our appetite judge was scoring class code heavily and total insured value barely, so it kept passing accounts that underwriters rejected on size. We could see that in a single query because human and judge scores live in one table with one shape.

The value of an eval is not its score. It is the gap between its score and a human's — and you cannot measure a gap across two schemas.

Judge-versus-human, as a standing metric

Once the comparison is cheap, it becomes a metric you watch rather than an investigation you run.

# Agreement between the judge and the humans, per eval, per week.
# A judge whose agreement is falling is a judge you can no longer trust
# to gate anything.

SELECT j.name,
       COUNT(*)                                        AS pairs,
       AVG(CASE WHEN SIGN(j.value - 0.5) = SIGN(h.value - 0.5)
                THEN 1.0 ELSE 0.0 END)          AS agreement
FROM   scores j
JOIN   scores h
       ON  h.target_id = j.target_id
       AND h.name      = j.name
       AND h.source    = 'human'
WHERE  j.source = 'judge'
GROUP BY j.name;

Notice the join is trivial — same table, same columns, different source. That is the entire argument for the unified primitive. In a two-schema world this query is an integration project, so it never gets written, so nobody ever finds out that the judge drifted three months ago.

Scoring business events, not just model outputs

The part that is unusual: a score can attach to a business event, not only to a model call.

That sounds like a small extension and it opens up a different class of question. "Was this reserve set at a reasonable level?" is not a question about a model output — it is a question about a number that ended up in the claims system, which may have been set by an agent, adjusted by a supervisor, and later revised. Scoring the event lets a reviewer assess the outcome regardless of how it was produced, and lets you compare agent-set reserves against human-set ones on the same axis.

In practice this is how we found that agent-set reserves clustered slightly high on small claims: the human reviewers' scores on reserve_adequacy were consistently lower for agent-authored events than for human-authored ones, in a band where the agent was over-reserving out of caution.

What this looks like in InsightLense

Because every score has the same shape, InsightLense can put all four sources on one row and show you where they disagree. This is the view the whole primitive exists to make possible.

Scores eval = appetite_fit judge/human agreement 0.62

Target	judge	code	human	feedback	Agreement
VS-2027-0832950	0.82	true	reject	—	disagree
VS-2027-0833104	0.44	true	reject	—	agree
VS-2027-0833220	0.91	true	accept	▲	agree
VS-2027-0833388	0.88	false	reject	—	disagree

Two disagreements with the same shape. Both are accounts the judge scored highly and an underwriter rejected — and both are above the preferred size band. Forty rows like this is a calibration finding; in two separate systems it is invisible.

InsightLense tracks judge-versus-human agreement as a standing metric per eval, so a judge that drifts shows up as a falling number rather than as a surprise months later. Because scores also attach to business events, the same view answers "were agent-set reserves scored lower than human-set ones" — a question about outcomes rather than about model outputs.

Practical advice if you are building this

One table, from the start. Merging three schemas later is a migration nobody schedules. The cost of the unified shape on day one is close to zero.
Record who scored, not just what. eval:appetite-v3 lets you find every score produced by a judge you later stop trusting. Without it, a bad judge poisons your history invisibly.
Do not average across sources. A mean of a human score and a judge score is a number with no meaning. Keep them separate and compare them; that is the whole point.
Booleans before numerics in shape detection. See the note above. It is a one-line bug with permanent data loss.
Let scores be sparse. Most runs will have a code score, some will have a judge score, few will have a human score. That is correct and the queries should expect it.

One primitive, four sources, three shapes. The design is deliberately unambitious — its value is entirely in the joins it makes trivial.

Where do your agents already act on real records?

Tell us that, and what you would need to prove about those actions to an auditor. We will set up a hands-on walkthrough within two weeks.

Request a demo →