We found the same bug five times in our own products. Every occurrence had passed code review. Every occurrence had a correct schema, a sensible type, an index, and in two cases documentation describing exactly what the field was for. In all five, nothing ever wrote to it.
It is worth naming this defect class, because it is invisible to every test you are likely to have and it is endemic to systems assembled from several services.
The shape
A field that exists everywhere except in the data.
It is declared in a model. It has a type. There is a migration that created the column. There may be an index on it, which means someone thought about query patterns. There is often a docstring. It appears in the API schema, so it shows up in generated client libraries and in your documentation site. Every artefact around the field says the field is real.
The field is always null.
The five
1. The Pydantic model that dropped what it was given
A trace ingestion endpoint. The caller sent a rich payload; the response model declared a subset. Pydantic did exactly what it was told and discarded the rest, silently, on both the way in and the way out.
# The caller sends this...
{"trace_id": ..., "name": ..., "prompt_version_id": ..., "correlation_id": ...}
# ...but the schema only declares this:
class TraceCreate(BaseModel):
trace_id: str
# name, prompt_version_id and correlation_id are not declared,
# so they are dropped. No error. No warning. No log line.
# The database column exists. It is never populated.What made this hard to see: the endpoint returned 200, the row was created, the UI rendered, and the only symptom was an empty column in a view nobody looked at yet.
2. The parameter that was always an empty list
An email-intake route accepted attachments, passed them to a processing service, and the service
persisted them. Except the route constructed the request object with attachments=[] and
the request schema had no attachments field to put them in anyway. Two independent breaks in the same
path, each of which would have been sufficient on its own.
Submissions arrived. Documents were referenced in the body text. The attachment table stayed empty for weeks.
3. The correlation id threaded by nobody
Three services, each with a correlation_id column, each with middleware that could
read the header. No service ever sent the header. Each was ready to receive a thread; none
started one.
4. The action log that declared four fields and accepted three
Same root cause as the first, different schema. The writer sent an authority outcome; the model did not declare it. Every agent action recorded the tool name and the result, and never recorded whether the action had been permitted.
5. The response model, in the other direction
The subtlest one. Data was written correctly. The database had it. The response model omitted the field, so every consumer of the API — including our own UI — saw null and concluded the data was not there. We nearly ran a migration to backfill data that already existed.
Why tests do not catch it
This is the part worth internalising. Look at what a typical test asserts.
def test_create_trace():
r = client.post("/traces", json=payload)
assert r.status_code == 200 # passes
assert r.json()["trace_id"] == expected # passes
# Nothing asserts that prompt_version_id survived the round trip,
# because the test author wrote the test from the same schema
# that is missing the field.The test and the bug were written from the same incomplete schema. That is why the test agrees with the bug.
Type checkers do not catch it either — the code is correctly typed, it is just typed to a shape that omits something. Integration tests do not catch it if they assert on the same fields the schema declares. And the runtime is entirely happy: dropping undeclared fields is the documented, intended behaviour of the validation library.
How we actually found them
Not through tests. Through a business ledger with an empty column.
flowchart LR
W["Writer
sends full payload"] --> S{{"Schema
declares a subset"}}
S -->|declared| DB[("Column populated")]
S -->|undeclared| X["Silently dropped"]
DB --> V["Ledger view"]
X -.->|nothing arrives| V
V --> H["Human notices:
this column is
always empty"]
style S fill:#fbede3,stroke:#b4531b
style X fill:#fbede3,stroke:#b4531b
style V fill:#e2f5f9,stroke:#0e97b0
style H fill:#efecfe,stroke:#4b34e0
The reason a business ledger surfaces this so quickly is that it is read by people who did not write the code, in a format where a blank column is obvious. A model-call log would not have helped: the model calls were all fine. The information was lost in the plumbing between the agent and the system of record, which is precisely the layer a model-call log does not cover.
Four defences that actually work
Reject unknown fields instead of dropping them
The single highest-value change. Configure your validation layer to error on extra fields rather than discard them, at least on internal service-to-service boundaries. A loud 422 during development is infinitely better than a silent null in production.
class TraceCreate(BaseModel):
model_config = ConfigDict(extra="forbid") # fail loudly
...Assert the round trip, not the status code
Write one test per boundary that sends a fully-populated payload and asserts that every field comes back. Generate the field list from the database model rather than the API schema — the whole point is that the two disagree.
Alert on always-null columns
A scheduled check: for each column that is nullable but expected to be populated, what fraction of rows in the last seven days are null? Anything at 100% is either a dead feature or this bug. We run this and it has paid for itself.
Treat an empty column as a defect, not a data gap
The cultural half. When someone says "that column is always empty", the instinct is to assume the feature is not in use. Check whether anything could write to it before you accept that explanation.
Defect 5 — the response model — was one review comment away from becoming a backfill migration against data that was already correct. Writing to a table to fix a read bug is how you turn a display problem into permanent corruption. Before you backfill, query the table directly and confirm the data is genuinely absent.
What this looks like in InsightLense
Every one of the five defects was found the same way: a human looked at a ledger view and saw a column that was always blank. InsightLense now runs that check on a schedule rather than waiting for someone to notice.
| Table | Field | Rows | Null | Status |
|---|---|---|---|---|
| llm_traces | model | 41,208 | 0.2% | ok |
| llm_traces | prompt_version_id | 41,208 | 100% | never written |
| agent_actions | authority_outcome | 8,116 | 100% | never written |
| submissions | attachments | 1,204 | 100% | never written |
This is a small feature and it has caught more real bugs for us than any other single check. The reason it works is that InsightLense sits at the point where data lands rather than where it is sent — every automated check upstream of the schema agrees with the schema, which is exactly why they all agreed with the bug.
The sibling pattern
There is a closely related defect that is worse, because the field is populated and everything looks healthy: a value that is computed, recorded, displayed on a dashboard — and never acted on. We had one in our renewal engine. It has its own post: the control that was computed but never enforced.
Both defects come from the same place. In a system assembled from several services, it is possible for every individual component to be correct and for the composition to do nothing. The only reliable way to find that is to look at what actually landed, in a form a person can read.
Where do your agents already act on real records?
Tell us that, and what you would need to prove about those actions to an auditor. We will set up a hands-on walkthrough within two weeks.
Request a demo →