The Label Arrived Two Years After the Vocabulary

Yesterday we showed that the caseType field of our own PRC judgment corpus is stratified by year, and that six classes in the court-registry vocabulary have no counterpart in the shorter vocabulary that surrounds it on both sides. We left the obvious question open: are those cases missing from the years that have no word for them, or is only the word missing?

Today we answered it, and the answer is worse than either option. The cases are all there — every one of the six classes is present in every era, and the court’s own case number says so. But the shortfall has three different shapes, only one of which is fixed by knowing which era you are in. And the finding that reorganised the day is this: the registry vocabulary switched on in 2020, and these six classes did not become usable until 2022. For property-preservation dockets in 2020, the class exists, the vocabulary is in use — and 0.2 per cent of the dockets carry it.

Six classes, one era, and a question with a checkable answer

The six registry classes with no counterpart elsewhere are non-litigation preservation review, jurisdiction, compulsory liquidation and bankruptcy, judicial sanction, international judicial assistance, and interregional judicial assistance. Together they carry 1,009,605 records under those labels.

Chinese courts assign every docket a case number containing a short registration code that names the programme the case was filed under. Preservation applications get CAI BAO (property preservation), ZHENG BAO (evidence preservation) or the administrative variant; jurisdiction disputes get MIN XIA and its criminal and administrative siblings; bankruptcy and liquidation get PO, PO SHEN, QING SHEN, QIANG QING; judicial sanctions get SI CHENG; assistance requests get XIE WAI REN for foreign awards and REN GANG, REN TAI, REN AO for Hong Kong, Taiwan and Macau. That code is written by the issuing court at registration and is independent of whatever classification our pipeline later stored. It gives us a second opinion for every record in the corpus, including all the years in which the class label does not exist.

The method, and the mapping error it caught in the first hour

Our first pass did this by sampling: draw the six labelled populations, read the case-number codes they carry, learn the code-to-class mapping from what the sample shows, then apply that mapping to random samples of each era. It produced a headline number of 5.23 million preservation dockets corpus-wide against 707,825 labelled ones — a factor of 7.4.

That number was wrong, and the way it was wrong is the reason we are describing the method at all. The learning sample of 2,994 preservation records contained exactly one record carrying the code ZHI BAO. On the strength of that single observation the code went into the mapping table as a preservation code. It is not one. Checked against every record in the labelled era rather than a sample, ZHI BAO appears on 2,470,248 dockets, of which 97.1 per cent are stored as enforcement cases. It is an enforcement-stage code that happens to share a character with the preservation family. One observation in three thousand had inflated the estimate by roughly 2.5 times, from an exact 2,106,487 to 5,229,881.

Two things follow. The first is a rule: a mapping entry supported by a single sampled record is not a mapping entry. The second is a better method. Because the case-number field is stored as an exact-match keyword, a substring query against it runs in about a second over the whole corpus — so the mapping does not have to be learned from a sample at all. It can be verified against every record in the era that carries labels, by crossing each candidate code with the stored class and reading the agreement rate. Every figure from here on is produced that way.

The two rows that disagree with their own label — CAI BAO at 54.6 per cent and SI CHENG at 43.8 per cent — are not evidence that the code is wrong. They are the finding, and the next section is about them.

What the case numbers find

Here is the whole corpus, by class, counted twice: once by the stored label and once by the court’s registration code. The recall column is what a buyer filtering on the class name actually gets.

No class is anywhere near complete, and no two are incomplete for the same reason. Read the last two columns together: bankruptcy loses nothing inside the labelled era and a quarter of itself outside it, while preservation loses nearly half of itself inside the era that supposedly labels it. Those are different defects wearing the same percentage sign.

Three shapes of loss

Shape one: era-limited, but accurate where it exists

Bankruptcy is the well-behaved case. Inside 2020–2024 the label catches 89,436 of 90,697 registered dockets, 98.61 per cent. Everything it misses sits in years that have no such class: 13,388 dockets in 2008–2017, 14,881 in 2018–2019, 2,609 in 2025, 573 in records with no usable year. Jurisdiction behaves the same way at 94.12 per cent inside the era, with 242,228 dockets stranded in 2008–2017 alone — more than the labelled era holds.

This shape is the one a crosswalk actually fixes. Derive the class from the case number for every year outside 2020–2024, keep the stored label inside it, and bankruptcy recall goes from 73.20 per cent to something close to complete. Nothing about the stored data has to change.

Shape two: the label existed and was not applied

Preservation and judicial sanction do not behave that way, and this is where the day’s headline comes from. Splitting the labelled era by year:

Yesterday we established that 2020 records overwhelmingly speak the registry vocabulary — 90.6 per cent of that year’s records carry one of its twelve classes. Both facts are true at once, and together they say something we did not expect: a vocabulary can be in use for two years before its finer classes are. In 2020, of 377,547 dockets registered under CAI BAO, 89.3 per cent are stored as “civil case” — the generic parent — and 0.2 per cent as preservation review. In 2021 the split is 56.2 per cent civil against 42.7 per cent preservation, a class visibly being adopted mid-stream. Only from 2022 does the specific class win.

So “which vocabulary does this year speak” is not a sufficient question. The right one is “which vocabulary, and which of its classes had started being applied by then” — and the answer differs by class. The 2024 dip to 71.4 per cent is the other end of the same story: it is the cutover we dated to 31 October 2024, arriving from the opposite direction.

The practical consequence is the reverse of the usual one. In 2020, a buyer filtering for civil cases receives roughly 337,000 preservation rulings they did not ask for — procedural orders freezing assets, not adjudications on the merits. A model trained on “civil judgments” from that year has been fed a third of a million documents of a different kind. The under-applied label does not only cost you recall on the rare class; it costs you precision on the common one.

Shape three: the label itself is impure

Interregional judicial assistance is the smallest class in the field and the only one where the label is wrong rather than merely absent. Of the 92 records carrying it, 59 carry one of the four recognition codes for Hong Kong, Taiwan or Macau judgments. The other 33 — 35.87 per cent — carry administrative state-compensation codes, first instance and appellate. They are not assistance cases at all. A 92-record class with a third of its contents misfiled is a rounding error by volume and a live hazard by use: it is exactly the kind of class someone builds a demo or an eval set from, precisely because it is small enough to read.

Compare the other five, where the label agrees with the case number on 96.31 to 100 per cent of the records it covers. The problem with those five is that they cover too little. The problem with this one is different in kind.

The field that sometimes holds an entire judgment

One more thing surfaced while validating the substring method, and it belongs to the record. Reading through case numbers matched by the bankruptcy code, one was not a case number: it was the complete text of a divorce judgment, some 1,900 characters, sitting in the case-number field. It matched because the word for “breakdown” in “breakdown of marital affection” shares a character with the bankruptcy code.

It is not unique. 40,080 records carry a case-number value of 60 characters or more; 19,843 of 120 or more; 1,706 of a thousand or more. A well-formed PRC case number is roughly fifteen to twenty-five characters. So somewhere upstream there is a path on which a field boundary is lost and the document body lands in the identifier column.

For today’s counts the contamination is small and we bounded it rather than assuming it: of the dockets matched by each code family, the share whose case-number value is 60 characters or longer is 0.01 per cent for preservation, 0.00 per cent for jurisdiction and sanction, and 0.22 per cent for bankruptcy — the worst case, because a single common character carries the match. That is the honest error bar on the bankruptcy row, and it is smaller than the effect being measured. It is also a reminder that any substring method over a corrupted field inherits the corruption, and the only defence is to measure how much.

How to reproduce this against any corpus in an afternoon

Nothing above needed a model, a labelled gold set or an annotator. It needed four queries, and any vendor whose records carry the court’s own case number can run them for you or let you run them yourself.

1. Pick the years in which the class label is populated at all. One terms aggregation of the class field bucketed by year gives you this. Do not assume the vocabulary’s start date is the class’s start date — that assumption is what this piece is about.
2. Verify each candidate registration code inside those years. Cross a substring query on the case-number field with a terms aggregation on the class field. A code whose top class is the one you expect, at ninety-something per cent, is a usable derivation rule. A code whose top class is something else — ZHI BAO at 97.1 per cent enforcement — is a trap you just avoided.
3. Count both populations across every year, including the ones with no label. Two counts per era: dockets matching the verified codes, and records carrying the label. Their ratio is recall; run it per year, not per era, or the 2020 collapse averages away.
4. Bound the identifier column. One length histogram, and the share of your matched dockets that sit in the malformed tail. Quote it next to every figure the substring method produced.

The whole run took under an hour of query time on our index. What it produces is not a quality score but a per-class, per-year recall table — the artefact that tells you, before a contract rather than after an evaluation, which of your intended filters will work and in which years. For four of the six classes here, the honest answer is that the filter works well from 2022 and should be replaced by a case-number derivation everywhere else.

It is also worth saying what the exercise cost in the other direction. The jurisdiction class strands 242,228 dockets in 2008–2017 and another 192,025 in 2018–2019 — together more than twice what the labelled era holds. Anyone studying how Chinese courts allocate jurisdiction, which is a procedural question with a decade of relevant material, would using the label alone see the most recent five years and conclude the phenomenon is recent. It is not. It is only recently spelled.

What we would hand a buyer

1. Ask for label recall per class per year, not per field. A field-level completeness figure would have reported this caseType field as well populated. The defect only appears when the class and the year are crossed, and it differs by class in the same year.
2. Never accept a mapping entry supported by one sampled record. Ours cost a 2.5-times overstatement and would have shipped as a headline number. Where the identifier column supports exact substring matching, verify the mapping against every record in the years that carry labels instead of learning it from a draw.
3. Treat a rare class as under-applied until shown otherwise, and check the generic parent first. When a specific class is missing, its records are usually sitting in the broader class one level up, which means the parent is contaminated in the same proportion.
4. Read the smallest classes by hand. The only class here whose label is affirmatively wrong has 92 records. No aggregate statistic would have caught 33 misfiled ones; opening them did.
5. Bound the identifier column before deriving anything from it. One length histogram over the case-number field is enough to turn “the case number is reliable” into a number you can quote.

What we are not claiming

We measured no other provider’s data, so nothing here compares vendors. The case number is a strong second opinion and not an oracle: it records the programme the issuing court assigned at registration, so a court that registers a case under the wrong code is copied faithfully by our check, and a docket can be re-registered mid-life. The substring method matches a code anywhere in the field rather than at its structural position, which is why the length bound above is stated rather than skipped. Era buckets use the stored judgment year, which for a minority of records differs from the bracketed filing year in the case number; a docket filed in 2019 and decided in 2020 counts here as 2020. The 33 misfiled records in the interregional class are described, not explained — we know what they carry and not how they got there. And nothing has been repaired: no label has been backfilled, no oversized case number truncated, because changing stored values without a change log breaks every join written against the current ones. All of it is fixable; all of it needs notice first.