The Fallback Is Thinnest Exactly Where You Need It

Yesterday we published a category-recall audit of our own PRC judgment corpus and closed it with a note to ourselves: now that a substring query on the case-number column returns in about a second over the whole index, every conclusion we had previously reached by sampling was cheap enough to re-run exactly. Three of those conclusions were load-bearing. We re-ran all three today.

Two moved. One held. And while re-running them we tripped over something that matters more than any of the three: the coverage figure we had been quoting for our case-number fallback is measured on the wrong denominator. Across the corpus the rule reads 93.33 per cent of records. On the records that actually need it — the ones whose stored field is blank — it reads 80.15 per cent. The reason is not subtle once you look: blank case numbers are 305 times denser in the slice that needs the fallback than in the slice that does not.

The rule, written out before the numbers

Chinese courts write a short registration code into the case number when a docket is opened. The code is assigned by the court, not by whoever later loaded the file into a database, which is what makes it a usable second opinion about a record whose stored metadata is missing or wrong. The rule we use throughout this piece asks one question: does the case-number string contain any of nine single-character stage codes?

That is the whole rule. It is worse than the parser we actually ship, which handles pre-2016 court-invented abbreviations and sub-docket suffixes. We are using the crude version deliberately: the same rule is applied to both sides of every comparison below, so whatever it gets wrong, it gets wrong symmetrically. A reader who prefers a better parser should expect both sides of each comparison to rise, and the gap between them to be the durable part.

Re-audit one: the bucket that moved by 111,492 records

Our case-type field has a residual value that translates as “other”. It holds 5,888,436 records. Twice we sampled it — 1,000 records, then 4,998 records with fixed seeds — and twice concluded the same thing: it is not a residual category at all, it is a missingness marker wearing a category’s clothes, because the majority of what sits in it is ordinary civil litigation that the short vocabulary had a perfectly good word for. The two samples put that majority at 59.3 and 58.20 per cent.

Counted exactly today, allowing the civil family codes to be separated from the stage code by an inserted character — which is how pre-2016 case numbers are written — the answer is 3,538,562 records, 60.09 per cent.

Every sampled figure was close. That is the point worth sitting with, because “close” is not the unit the buyer works in. The 58.20 per cent estimate corresponds to 3,427,070 records. The exact count is 3,538,562. The difference is 111,492 judgments. A percentage that is right to within two points is a number that is wrong by six figures, and six figures of judgments is the unit a licence is written in, a training run is budgeted in and an evaluation set is drawn from.

The families barely overlap, which is the check that makes the table readable rather than merely arithmetically valid: 75 records carry both a civil and an enforcement code, and 264 carry both a civil and a criminal code. Out of five and a half million assignments, that is the collision rate you want before you present shares that sum.

One sub-finding is worth its own line, because it is a measurement of a parser rather than of the data. Requiring the civil code and the stage code to be adjacent finds 3,198,349 records. Allowing one inserted character between them finds 3,257,124. The difference, 58,775 records, is entirely pre-2016 docket formatting. One character of tolerance in one regex is worth fifty-eight thousand judgments inside a single bucket.

Re-audit two: from “mostly right” to “trust it”

The procedure field in our corpus carries four mutually unintelligible encodings, one of which is a set of bare single digits. From a ten-thousand-record random sample we had reported that digit 1 is first-instance about 96 per cent of the time and digit 2 is appellate about 96 per cent of the time. Both numbers were honest summaries of the sample. Both were wrong about the code in ways that change what you do with it.

Denominators for those agreement rates are the records in each code whose case number is non-blank: 99.87, 99.86 and 99.67 per cent of each code respectively, so almost nothing is being hidden by the denominator choice.

Code 2 is the one that matters. Ninety-six per cent and 99.59 per cent are on opposite sides of an operational line. At 96 per cent, a team building an appellate-only evaluation set has to sample and hand-check, because one item in twenty-five is something else. At 99.59 per cent, the same team can take the code at face value and spend its annotation budget elsewhere. The sample did not merely under-report the code; it under-reported it into a different decision.

Re-audit three: the one the sample got right

The third estimate was the ugliest of the three, and it survived. Our earlier work warned that the stored phrase meaning “retrial” is an umbrella term, and that mapping it to the retrial code in a crosswalk would be wrong for roughly half the records it covers. That warning was based on a sample of the 139,409 records carrying that value.

Exactly: 46.02 per cent carry an actual retrial code, 46.56 per cent carry a retrial-application code, 2.05 per cent an appellate code and 1.10 per cent a first-instance code. The distinction is not pedantry. A retrial-application docket is a court deciding whether to reopen; a retrial docket is a court that has reopened. Any crosswalk that flattens the two produces an eval set where roughly half the “retrials” never happened.

We record this one deliberately. Two of three sampled conclusions moved and one did not, which is exactly the pattern that makes sampling dangerous rather than useless: you cannot tell from inside the sample which kind you are holding.

The finding that reorganised the day

Our procedure field is blank on 21,393,940 records. That is the slice the case-number fallback exists for. So we ran the same nine-marker rule on both sides of that boundary.

The corpus-wide 93.33 per cent is arithmetically true and operationally useless, because it is dominated by records that never needed the fallback. Conditioned on need, the rule loses 15.21 points, and it loses them almost entirely to one failure mode: the input the rule depends on is itself absent. Blank case numbers run at 0.05 per cent where the procedure field is populated and 16.06 per cent where it is blank — a density 305 times higher in exactly the place the repair was supposed to happen.

Of the blank-procedure slice, 80.15 per cent is recoverable, 16.06 per cent has no case number to read and 3.79 per cent has a case number our rule cannot parse. Those three numbers want three different responses, and a single coverage figure hides all of them.

Why the fields fail together

None of this is coincidence, and the cause is checkable in one query. Our corpus holds 3,509,386 records with a blank case-number value. Of those, 3,436,255 — 97.92 per cent — also have a blank procedure value. And 3,438,131 of them come from a single ingest batch; the next largest contributor supplies 71,249, and two other batches supply six records between them.

So missingness here is not a property of individual records. It is a property of a batch, and fields inside a batch fail in company. This is why a whole-corpus completeness figure for field A and a whole-corpus completeness figure for field B, published side by side, can both be reassuring while the joint distribution is not. The cross-tabulation is the number; the two marginals are decoration.

It also predicts a specific, unpleasant behaviour at query time. Those blank case numbers are not distributed evenly across subject matter either: 1,843,924 of them sit under the civil case type, 921,041 under the residual “other” type and 523,509 under enforcement. A buyer who filters to civil cases and then measures how often the fallback fires will get a different answer than a buyer who filters to criminal cases, without either of them doing anything wrong.

An exact count still has a residual, and it is yours

The last re-audit is the one that cost us the most self-regard, and it is the reason the nine-marker rule is printed in full at the top of this piece.

Digit code 4 covers 9,641,805 records. Under the nine-marker rule, 78.43 per cent are enforcement and a further 1.38 per cent are special procedure — leaving 1,924,918 records, 19.97 per cent of the code, which have a perfectly good case number that the rule simply could not read. It would have been easy, and wrong, to publish that as a data-quality finding.

It is not one. Reading the residual instead of counting it:

• 891,067 records (46.29 per cent of the residual) carry XING-GENG, the sentence-modification code — parole, commutation and similar post-judgment criminal dockets.
• 391,125 (20.32 per cent) carry XING-SHEN, administrative non-litigation review, where a government body asks a court to enforce its own decision.
• 388,857 (20.20 per cent) carry BAO, property preservation.

Adding those three codes and a compensation code to the rule takes the genuinely unexplained share of code 4 from 19.97 per cent to 2.61 per cent. Four rules, 17.4 points, in one code. And the substantive correction is that code 4 is not an enforcement code at all: it is an omnibus bucket for everything that is not an ordinary trial stage, which is a different thing to tell a buyer and a different thing to build a filter on.

This is the third distinct instance we have logged of a single parser rule being worth double-digit percentage points, after a sub-docket suffix rule worth 6.8 points and a pre-2016 commercial-code rule worth 25.8 points in one era. The returns are visibly diminishing and the curve has not flattened. The honest way to describe a case-number parser is as an asset under maintenance, not a solved problem.

Running this check on a corpus you have not bought yet

Everything above is four queries. A buyer in a trial window can run the same four against any vendor’s sample extract, and the answers are diagnostic in a way that a completeness table is not.

1. Count the blanks in the field you care about, then count the blanks in the field its fallback reads. Separately, these are two reassuring numbers. Then cross-tabulate them. If the joint count is close to the smaller of the two marginals, the fields are failing together and the fallback is structurally unavailable where you need it. In ours the joint count is 97.92 per cent of one marginal, which is about as correlated as two fields get.
2. Ask for the derived-field coverage figure again, restricted to records where the stored value is absent. If the vendor cannot produce it, they have not measured the thing they are selling. If the number comes back materially below the headline — ours drops 15.21 points — that gap, not the headline, is what your pipeline will experience.
3. Group the blanks by ingest batch. If they cluster, every coverage number you were quoted is a weighted average over batches whose weights will change the moment you apply any filter. A corpus where one batch supplies 97.97 per cent of a defect is a corpus where “exclude that batch” is a real option and should be priced as one.
4. Demand the residual split three ways: no input at all, input present but unparseable, and input the rule was never taught to read. Only the first is a hard limit. The second is a parser backlog. The third is not a data defect and should not be sold to you as one, nor deducted from the vendor’s credibility when it is disclosed.

Run in that order, the four questions take under an hour on a sample extract and they are hard to answer evasively, because each one is a count rather than a characterisation. The vendor who has already run them will have the answers written down. The vendor who has not will offer you a field-level completeness table instead, and a field-level completeness table would have described everything in this article as healthy.

What we changed about how we publish numbers

Five rules came out of today, and we are writing them down because all five are things we had been getting wrong in public:

1. Coverage of a derived field is reported conditioned on need — measured on the records whose stored value is absent, never on the whole corpus.
2. Completeness is published as a cross-tabulation between the field and the input its fallback depends on, because the marginals conceal the correlation.
3. A percentage from a sample is accompanied by the record count it implies, since two points of sampling imprecision is six figures of judgments here.
4. A residual is labelled by kind — no input, unparseable input, or rule blind spot — and the last of those is never presented as a data defect.
5. Any rule used to produce a published number is printed with the number, at the granularity that lets a reader predict what it will miss.

What we are not claiming

We measured no other provider’s data, so nothing here is comparative. We fixed nothing: no field was backfilled, no batch was quarantined, no stored value was rewritten, because changing stored values breaks every join already written against them and that belongs in a change log with notice, not in an audit. The case number is a strong second opinion, not an oracle — it records the procedure a court assigned at filing, and a court that registered a docket wrongly will be faithfully reproduced by this check. Substring matching asks whether a code appears anywhere in the field, not whether it appears in the structurally correct position, which is why the case-number length distribution matters and why we published it yesterday. And the three re-audited estimates were re-run against the same index by the same team that produced the originals; this is an internal consistency check, not independent replication.

One number from today has no comforting reading. Even under a better parser than the one used here, roughly one record in six in the blank-procedure slice has no case number at all. For those, there is no derivation, no fallback and no clever rule — only the upstream pipeline that dropped the field, and a change log honest enough to say so.