Derived, Not Observed: Auditing a Metadata Field That Fills Itself

Yesterday we published an audit of our own province labels and concluded that the case number is the one field on a Chinese judgment that no pipeline invented. Today we turned the same instrument around, because there is a second half to that story that the first half cannot answer.

Our public key-free case endpoint does something worth knowing about before you count anything it returns: when the upstream province is empty, it computes one at read time from the regional character in the case number, and hands you the result in the same field, in the same shape, as a province that was actually stored. We disclose the derivation in the response — each such record carries province_backfilled, province_backfill_kind and province_backfill_abbr. What we had never done is ask the obvious follow-up question: is the computed value right, and could anybody tell if it were not?

Across 2,522 records pulled on 4 September 2026, 141 of them — 5.59 per cent — carried a province we computed rather than read. Every one of those we could check against an independent field agreed. And 53.9 per cent of them cannot be checked against anything else in the record at all. That second number is the finding. A derived value that nothing in the record can contradict is not a measurement, it is a well-formed assertion, and it will be counted as a measurement by anyone who does not know the difference.

Why the case number cannot referee this

The temptation is to audit the computed province the way we audited the stored one: read the regional character out of the case number and compare. That is circular. The regional character is the input to the derivation. Comparing the output to the input measures whether our lookup table has typos, not whether the record is in the right province.

There is exactly one other place in a Chinese judgment record that states a location on the court’s own authority: the name of the issuing court. Chinese courts are named after the administrative unit they sit in, and higher courts name the province outright — 湖南省高级人民法院 announces Hunan in its first three characters. That name is transcribed from the judgment, not computed by us, so for this audit it is an independent referee — where it exists, and where it names something at province level.

So the audit is not “how often is the fill wrong.” It is two questions in order: how often is a referee even present, and when one is present, does it agree. Vendors, including us, reflexively answer the second question and quietly skip the first. The first is where the damage lives.

Result: right wherever it can be checked

Of the 141 records whose province we computed at read time:

Sixty-five out of sixty-five is a clean result and we are not going to dress it up as more than it is. It says the lookup from regional character to province is not silently broken on the slice we walked. It says nothing about the 77 records — 53.9 per cent — where the only two statements of location in the record are the case number and a value derived from the case number.

The not-checkable class is not exotic. It is ordinary basic-level courts: 杭州市上城区人民法院, 洛阳市西工区人民法院, 六盘水市水城区人民法院. Every one is resolvable by a human with a gazetteer, and none is resolvable by the record. It also contains cross-jurisdiction courts whose names deliberately span two places — the Chengdu–Chongqing Financial Court carries both cities in its name and sits in one of them — and railway transport courts, which are named for a railway bureau rather than a region.

The 24 records with no court name at all are the sharper case. There the corpus offers a case number, a derived province and no other geography. If our table were wrong for one regional character, those records would be wrong, would look exactly like the right ones, and no query against that endpoint would reveal it.

The mirror-image failure: declining where the answer is next door

Forty-seven records in the same sample had an empty upstream province and did not get one from us: 36 where the case number offered no parseable regional character, 9 where the character was outside our table, 2 where the character deliberately maps to no province — the Xinjiang Production and Construction Corps code, whose courts sit inside Xinjiang but number under their own character, and which we refuse to merge into the province rather than invent a tidier answer.

Six of those 47 have a court name that states the province in plain characters. The derivation never looked.

Read those four rows together and the pattern is not a missing lookup entry, it is an architectural one. The derivation reads one field. The answer is in another field, in the same record, in the same response. A judgment whose court is named “Sichuan Province Nanchong City Gaoping District People’s Court” is filed with no province because a bracket fell off its case number.

The last row is worse than a gap, it is a mislabelled gap. Our own definition of that bucket is “the character does belong to a province, we simply do not have it in our table.” The character in question is the numeral two, from Tianjin No. 2 Intermediate People’s Court. It is not a regional character at all. Our classification asserted something false about it, in a machine-readable field, which is exactly the failure mode this whole article is about: a pipeline reporting a confident category where the honest answer was “this is not the kind of thing I parse.”

The 36 records in the unparseable bucket are worth one line of their own, because most of them are not judgments in any useful sense: they carry no case number, and the court field holds a sentence from the Supreme People’s Court rules on internet publication explaining why the document was not published. They count toward result totals, they occupy pagination slots, and they contain no court, no province and no text. If your coverage denominator does not exclude them, it is inflated.

The referee itself is not clean

Having made the court name the referee, we checked the referee. In the same 2,522 records, 892 distinct court values appear, and three of them carry an invisible bidirectional control character — U+200F RIGHT-TO-LEFT MARK — sitting in front of the first Chinese character. In bytes:

court: "\u200f沈阳市沈北新区人民法院"
utf-8: e2 80 8f  e6 b2 88 e9 98 b3 e5 b8 82 ...
        ^^^^^^^^ RIGHT-TO-LEFT MARK, renders as nothing at all

The damage is not that the character is there. It is that all three of those court names also appear in the same sample without it:

Group by court name and each of those three courts becomes two courts, with identical labels, in the same result list. Join a caseload table on court name and the two halves never meet. Diff two exports and the rows look unchanged because your terminal prints them identically. One further record in the sample carries a stray backtick in the same position. Nine records in total sat on affected values — and all nine came from one half of our sweep, with zero in the other, which is precisely why we are reporting it as a defect that exists rather than as a rate.

A one-line check, worth running against any name field in any corpus you are evaluating: count the values whose first character is not a Chinese character. It cost us five seconds and it is the only reason we found this.

Reproduce it

Both commands are against the public key-free endpoint and need no credentials. The first returns records whose province was computed at read time, with the provenance bits attached:

curl -sG https://tob.wenshucha.com/api/cases/search \
  --data-urlencode 'q=执行' --data-urlencode 'pageSize=50' \
  | jq '[.cases[] | select(.province_backfill_kind)
         | {case_no, court, province, kind: .province_backfill_kind}]'

The second counts, on one page, how many of those computed provinces have a court name that could contradict them:

curl -sG https://tob.wenshucha.com/api/cases/search \
  --data-urlencode 'q=破产' --data-urlencode 'pageSize=50' \
  | jq '[.cases[] | select(.province_backfilled == true)]
        | {computed: length,
           no_court: [.[] | select(.court == "" or .court == null)] | length}'

Page-level counters travel in the response as well: province_backfilled_count, province_backfill_withheld_count and province_empty_count. Two sibling endpoints emit the per-record bits without the page totals, and the two bulk-browse endpoints do not run this layer at all — there the province is the upstream value or nothing. That asymmetry is deliberate and it is the reason the same judgment can carry a province on one route and none on another. If you reconcile two of our endpoints and find that difference, it is this, not a data inconsistency.

The gate that fired zero times

The derivation has a safety mechanism we built for a specific hazard: some regional characters are ambiguous between administrative levels. The character for Qinghai Province is the same character that Shanghai’s Qingpu District Court uses in its own case numbers, so a naive lookup would file Shanghai bankruptcy cases in Qinghai. When the derivation recognises a character but cannot settle the ambiguity, it is supposed to decline and mark the record withheld rather than guess.

In 2,522 records that gate fired zero times. So did the withheld counter on all 54 pages.

We are reporting that as an absence of evidence, not as a clean bill of health. A guard clause with no observations in a sample this size is untested in the traffic we actually serve; it is not demonstrated to work. The honest reading is that ordinary queries do not surface the ambiguous characters often enough to exercise it, which also means a buyer sampling casually will not exercise it either — and the collision it guards against is exactly the kind that produces a small number of confidently wrong records rather than a visible outage. Ask vendors which of their safeguards have never triggered on real traffic. It is a more revealing question than asking which ones exist.

What a derived province costs you downstream

The reason to care about 53.9 per cent rather than 0 disagreements is that province is rarely a display field. It is a partition key.

Three concrete failures follow from treating derived values as observed ones. Stratified training sets: if you sample judgments proportionally by province to balance a model, a systematically wrong regional character does not add noise, it moves a block of records from one stratum to another, and every downstream evaluation inherits the shift while reporting balanced coverage. Jurisdiction-scoped retrieval: a product that promises “cases from this province’s courts” is making a claim about provenance, and a derived value that no field in the record can contradict is not provenance, it is a default. Due-diligence counts: when a coverage table says a province holds a given number of judgments, ask what share of that number was computed at read time. In our sweep the answer was between 4.5 and 6.5 per cent depending on the queries, which is small enough to ignore in a headline and large enough to matter in a per-province cell — the small provinces are where derived values concentrate, because they are where the ranker returns fewer well-formed records.

The fix in every one of those cases is the same, and it is not better inference. It is carrying the provenance bit through your own pipeline so that a derived province can be excluded from a count, flagged in a citation, or weighted differently in a sample. If a vendor hands you a value with no provenance, your pipeline cannot do any of that, no matter how good your own engineering is.

What we did not measure

• Any other vendor. Every defect above is ours. We ran nothing against anyone else’s corpus and make no comparative claim.
• A random sample. Ranked page walks over-represent whatever the ranker likes. The 5.59 per cent derived-province rate is a property of this sweep, not of the corpus; the everyday-query half returned 4.54 per cent and the enforcement-heavy half 6.46 per cent, which is the size of the swing you get from choosing different words.
• Whether the 77 unfalsifiable records are correct. We measured that nothing in the record can settle it. Settling it needs an external court-to-region table, which is a different exercise with its own error modes.
• Pre-2016 judgments as a class. The regional character only became a reliable provincial code with the 2016 case-numbering reform, and the derivation inherits that boundary. Older records fail into the declined bucket rather than being filled wrongly, which is the right direction to fail, but it means the derived-province rate is not evenly distributed across years.
• Any repair. Nothing was fixed today. This is a measurement, not a release note. When the six recoverable records and the bidi characters are dealt with, the fix date goes in our change log and the numbers in this article stay as published.

Five questions for any derived field, ours included

1. Which fields are computed, and does the record say so? If a vendor cannot tell you per record whether a value was stored or inferred, then every coverage number they quote mixes the two, and so will yours.
2. What is the referee, and how often is it present? Ask for the share of derived values that some other field in the same record could contradict. Accuracy on the checkable subset is the easy half; the unfalsifiable share is the number that decides how much of the field you can rely on.
3. Does the derivation read every field that carries the answer? Ours reads the case number and stops. Six records in a 2,522-record sample had the province written out in the court name and got no province.
4. Do the “unknown” buckets say something true? There is a real difference between “this belongs to a region we cannot resolve”, “this belongs to no region” and “this is not the kind of string I parse”. A vendor who collapses those three is asserting things they did not test.
5. Have the name fields been checked for invisible characters? Before any group-by or join on a court, party or region name, count the values that do not begin with the script you expect. Identical-looking labels that never match are the hardest class of error to see and the cheapest to find.

None of this is unique to us and none of it is unique to Chinese court data. It is what happens to any corpus assembled from documents that were written to be read by people and are now being counted by machines. The reason we publish our own numbers rather than a capability list is that a buyer can check these in an afternoon, and a capability list cannot be checked at all.