The Crosswalk You Cannot Finish

Yesterday’s piece ended on a promise. Having measured what happens when copies of the same judgment disagree, we wrote that most of the apparent disagreement was vocabulary rather than contradiction, and that this “needs a crosswalk, written once”. That is the kind of sentence vendors use to make a defect sound like a chore.

So today we wrote it. We enumerated every distinct value in the procedure field of our own PRC judgment corpus and tested each one against a reference the data carries internally: the procedure code inside the case number on the same record. The result is not the tidy lookup table the word implies. It is 405 rows in four mutually incompatible encodings, of which 104 can be written with confidence and 26 cannot be written at all — and those 26 cover more than a fifth of the records. The conclusion is not that our crosswalk is big. It is that the crosswalk is the wrong instrument, and the field should be derived rather than reconciled.

What is actually in the field

The procedure field is supposed to say what kind of proceeding produced the document: first instance, appeal, retrial review, enforcement. Across our corpus it holds 405 distinct values, and sorting them by shape rather than meaning already tells you the crosswalk will not be a rename.

Three encodings and a hole. The phrase vocabulary and the four-digit code both distinguish civil first instance from criminal first instance; the short numeric code does not. That asymmetry decides most of what follows, because a crosswalk can only be written at the granularity of its coarsest side.

The sixteen values in the last row are the cheapest possible test of whether anyone has ever looked at this field. One is a fragment of JavaScript source — the eight characters ion(){$(, the tail of a function(){$( declaration — and it is the stored procedure value of 14,363 records. Another reads, in full, “1 = first instance”: the legend of the numeric code list, leaked into the data it was meant to explain, on 15 records. A third is the word “test”, on 31 records; four more are bare punctuation; five are criminal sentencing factors that belong to a different field.

Together the sixteen cover 14,515 records, so none of it is load-bearing. But the legend is worth keeping: it is the only documentation of the numeric scheme anywhere in our data, and we found it by accident while counting junk.

Which batch speaks which language

Encodings do not mix evenly. By ingest batch, the field is close to monolingual — with two instructive exceptions.

Batch 2 is the whole of the numeric scheme: 39,341,140 records, thirteen values, no phrases. Batch 7 — the same batch whose court field contradicts its own case numbers on a third of testable records — has an empty procedure on every record. Batches 5 and 6 are internally bilingual, carrying phrases on 93 to 94 per cent of records and four-digit codes on the rest, which kills the convenient assumption that vocabulary can be assigned per batch. It has to be assigned per record.

The crosswalk, and how much of it can be written

To test a value without trusting any other stored field, we used the case number. A PRC case number issued under the 2016 numbering reform carries an explicit procedure code — MIN CHU for civil first instance, MIN ZHONG for civil appeal, ZHI for enforcement, XING GENG for a sentence adjustment. We have used the same string as a province validator; here it validates the procedure. For each of the 405 values we drew up to 500 records at random and asked how concentrated the parsed codes were.

Read the two ends together. 228 of the 405 values — more than half the crosswalk’s rows — are too rare to verify, and together they cover just 1,835 records. 263 appear on 100 records or fewer; 110 appear exactly once. A crosswalk is priced by rows, a corpus by records, and those two numbers point in opposite directions: most of the writing effort goes to values that will never change an answer, which is exactly where a tired analyst guesses.

Tier A is also smaller than it looks, because it is mostly synonym folding. Its 104 values collapse onto only 44 distinct case-number codes, 24 of which are reached by more than one spelling. The sentence-adjustment code is reached by eleven values across three encodings; the special-procedure code by thirteen. Two values differing only in word order — “restore enforcement” and “enforcement restored” — carry 245,710 and 179,791 records and mean the same thing.

The four-digit code is the best-designed vocabulary in the corpus

It is also the rarest. The 44 four-digit values are structured — first two digits a case family, last two a procedure within it — and decoded against the case numbers of the records carrying them, the families come out clean.

This is a real code list, close to the official case-family taxonomy, and the only encoding here from which both the family and the procedure can be recovered without consulting anything else. It covers 843,268 records. The phrase vocabulary covers vastly more and reaches the same expressiveness only by convention, and only when the phrase is one of the verbose ones.

What the numeric scheme actually encodes

The leaked legend told us 1 means first instance. The case numbers confirm it and add what the legend omitted: among records carrying 1 whose case number is testable, 96 per cent are first-instance proceedings, split across civil, criminal and administrative. 2 is appeal on 96 per cent, again across all three families. The numeric scheme is a clean one-axis vocabulary: it encodes the instance and says nothing about the case family.

That makes the crosswalk one-directional. A phrase always maps to a number; a number maps back only with a second field, and the field it needs — case type — is exactly the one that in this batch stores something else entirely. This is why 33.5 per cent of sampled records sit in tier B: not because the value is dirty, but because it is honestly coarser than the vocabulary you want to express it in.

Below the four codes that carry mass, the scheme falls apart quietly. 3 is a genuine ambiguity rather than an error: it covers both the review of a retrial application and the retrial itself, and no class reaches 80 per cent. The remaining 45 numeric values carry between one and 3,716 records each and come from a different batch with a different meaning again — there 5 is non-litigation administrative review on 94 per cent of records and 7 is sentence adjustment on 99 per cent. Two numeric schemes, no version marker, one field.

The values that cannot be mapped at all

The pattern is not randomness. Each of these is a true umbrella term: a word the source system used correctly for a group of proceedings the case-number vocabulary keeps apart. “Enforcement implementation” covers first, preservation and restored enforcement; “retrial” covers both reviewing a retrial application and conducting the retrial; 4 covers enforcement and sentence adjustment, both post-judgment work.

You cannot fix an umbrella term with a lookup table, and pretending otherwise is worse than leaving it alone: a row mapping “retrial” to the retrial-review code would be wrong on 51 per cent of the records it touches, silently, with no null to warn anyone. That is where a well-meaning normalisation pass does more damage than the mess it replaces.

The other field: correct, and still unusable

The companion field, caseType, fails differently. Checked against the case number on the same record it is right: agreement ran from 96.3 to 99.5 per cent across the four batches we tested. Its 32 values are nevertheless three word-lists.

• The long form — the official eleven-category taxonomy: “civil case”, “enforcement case”, “non-litigation preservation review case”, and so on.
• The short form — six values: civil, criminal, administrative, enforcement, compensation, other. It cannot express eight of the eleven categories the long form names.
• The document type — in one batch of 39,341,140 records the field holds judgment, ruling, mediation, notice or decision instead of a case category. A query for civil cases returns nothing from that batch, and not because it has no civil cases.

Then the trap. The short form’s “other” value carries 5,888,436 records, which reads as the residual bucket for categories the short vocabulary cannot name. We drew 1,000 at random and read the code out of each record’s own case number: 59.3 per cent are ordinary civil cases, mostly civil first instance — a category the short vocabulary names perfectly well. 17.3 per cent are enforcement, 3.6 per cent criminal; only 17.9 per cent had no readable case number.

So “other” is not a category. It is a missing-value marker that got a category’s name, and any evaluation filtering it out to tidy a distribution discards millions of ordinary civil judgments while believing it removed noise.

What to do instead: derive, don’t reconcile

The stored procedure value is a compressed, versioned, occasionally corrupted account of something the record already states exactly — in a string every court is obliged to get right, because it is how the court finds its own file. So stop reconciling the field. Read the case number, and use the field only where the case number cannot be read.

Two checks make that table trustworthy rather than merely convenient. Where both routes fire and the stored value is tier A, they disagree on 0.46 per cent of records, so the crosswalk is not fighting the case number where it applies. And the residual is diagnosable: 203 of the 272 residual records have no case number at all, and most of the rest are real case numbers that a single-case regex cannot read — consolidated enforcement orders covering a range of dockets, plus a few records where a fragment of stylesheet markup landed in the case-number field, the same scraper failure that produced the JavaScript above.

One implementation detail is worth more than it sounds. A parser expecting a case number to end in digits reads 90.4 per cent of our sample; teaching it the sub-docket suffix Chinese case numbers may carry — the equivalent of “no. 490, part three”, present on 6.84 per cent of records — raises that to 97.2 per cent. Seven points of coverage behind one regex, and a vendor who has not fixed it will report a lower ceiling for this technique than it has.

The pre-2016 half needs a second parser, and here honesty costs something. Old-format case numbers do encode the instance, but through court-local abbreviations rather than a national code list: in one 800-record sample we saw the same civil-first-instance idea written dozens of ways, each prefixed by a court’s own contraction of its name. 97.7 per cent of those prefixes still yielded an instance, but the vocabulary is open-ended by construction, so this parser is never finished. It is a maintained asset, not a solved problem.

The protocol, compressed

1. Derive procedure and case family from the case number, with one parser per numbering era and a sub-docket-suffix rule.
2. Use the stored value only as fallback, and only for values in tiers A and B.
3. Never map a tier-C or tier-D value. Emit a null and count it — an umbrella term mapped to its most common member is the error class that leaves no trace.
4. Treat “other” as missing, not as a category, in any vocabulary where you have not proved otherwise on a random sample.
5. Publish the residual. Ours is 2.72 per cent of a 10,000-record random sample, and its composition, not its size, is what a buyer should ask about.

A measurement note that changed twelve verdicts

Our first pass took the first 500 records the index returned for each value. That is not a sample; it is a walk through whichever segment happens to be first, and in a corpus assembled from batches, segment order correlates with batch. Re-running with an explicit random seed changed the tier of twelve of the 405 values, one of them completely: “non-litigation enforcement review” looked like a clean 89 per cent match to property preservation in the ordered sample, and was 71 per cent administrative non-litigation review — a different family — in the random one. Publishing the first pass would have meant a crosswalk row wrong for the plurality of its records.

This is the third time in ten days that an ordered read produced a plausible, publishable, wrong number here. The rule now applies without exception: every per-value composition figure is drawn with a stated random seed, and every ratio is computed with the total-hits cap disabled. As a cross-check, the shape shares measured by aggregation over every record (61.6 / 24.6 / 13.3 / 0.5 per cent) and over the 10,000-record random sample (61.5 / 24.3 / 13.9 / 0.4) agree to within half a point.

What we are not claiming

We measured no other corpus, so nothing here ranks anyone against anyone; every defect described is ours. The case number is a strong validator, not an oracle: it records the procedure the issuing court assigned, so a mis-docketed matter carries the error through, and it cannot check a cause of action at all. Tier assignments are sample-based, so the 95 per cent boundary will move a handful of borderline values under a different seed — which is why we publish the boundary and not only the verdict. And we have changed nothing: normalising this field would alter results for anyone already joined against current values, so it belongs in a change log with notice, not a quiet overwrite. Today’s output is a measurement and a protocol, not a migration.

For anyone evaluating a Chinese case law corpus, including ours, this reduces to four questions that are cheap to ask and impossible to answer vaguely. How many distinct values does your procedure field hold, in how many encodings? Which are umbrella terms, and what do you emit for those? Does your case-type field ever hold a document type instead? And what fraction of records can you assign a proceeding to without trusting the stored field at all? We have answered all four on our own data, failures included.