The Court That Stopped Hearing Cases in 2017

Yesterday we audited a field our API computes at read time, and we used the issuing court’s own name as the referee, on the grounds that the name is transcribed from the judgment rather than invented by us. That audit came back clean wherever it could be checked. It also left an obvious debt: we never asked whether the referee itself is sound.

Today we asked. The short answer is that the court name field passes every test we currently publish about it, and it is still wrong in a way that will corrupt the first analysis most buyers run on it. Not wrong record by record — wrong between records, which is the class of defect that record-level validation is structurally unable to see.

The finding in one sentence: the same court is carried under two different names by two different ingest generations, and grouping by name splits it into two courts that each appear to exist for only part of the corpus’s time range.

First result: the field is clean, and that is the problem

Our records carry a flag, court_is_court_name, and our own field notes tell buyers to filter on it before grouping by court, because the court position sometimes holds a non-publication reason rather than a court. Here is what that flag reports across the three cohorts.

Read the first row again. A buyer who evaluates this field by browsing measures a defect rate of exactly zero, on nine hundred records, and is not being deceived. The measurement is correct. It is simply a measurement of a different population from the one their product will query, because damaged records are short, thin documents that relevance ranking only lifts to the top when the query happens to touch what little text they contain.

Ninety-four records in the sample hold something other than a court name. Only five distinct values are involved, and four of them are non-publication reasons lifted verbatim out of the judgment portal: the category for divorce proceedings and matters touching the custody of minors (36 records), the category for juvenile offences (11), the residual category for matters a court considers unsuitable for internet publication (4), and cases closed by mediation (3). The fifth is a case title that landed in the court position. These records are also missing other things: 62 of the 94 have no case number and 54 have no judgment date. They are placeholders standing where a document was withheld, and they occupy positions in your result pages and in any total you compute.

Second result: the flag inspects the end of the string

We reimplemented the flag independently and compared. It agreed on 3,550 of 3,550 records. That sounds like a validation and it is not one — both implementations apply the same conservative rule, which is whether the value ends in the characters for court or tribunal. Two implementations of one rule agreeing tells you the code is faithful to the rule. It tells you nothing about what the rule cannot see.

What the rule cannot see is the front of the string. Fifteen records in the sample carry a court value that begins with something that is not a Chinese character, and every one of them is flagged as a valid court name:

These are not rendering artifacts of our analysis. We refetched the exact pages with a plain HTTP client and inspected the raw response bytes: the three-byte UTF-8 sequence for U+200F is present on the wire, unescaped, immediately before the first character of the court name. The value that begins yu and a newline was found on a page of the no-query cohort — the same cohort whose page-level counters reported zero empty court values and zero non-standard ones. The clean control sample contained a corrupted court name the whole time. Our meter and this defect simply do not overlap.

The consequence is not cosmetic. Six court names in this sample appear in both forms at once, marked and unmarked, and the two forms are visually identical in every terminal and browser we have. One basic-level court appears twelve times clean and once marked; another ten times clean and three times marked. A grouping operation reports two buckets. A join matches one of them.

The mark is a document-level artifact, not a field-level one

Pulling the full text as well, 25 records in the sample carry U+200F inside the body of the document, not merely in the court field — a median of 62 occurrences per affected document, with a maximum of 244, and 15.9 per cent of the 2,649 marks sitting at the start of a line. Twenty-four of those 25 records come from a single province and one from a specialised court system. Only 13 of the 25 have a mark that reached the court field. So this is not a court-name problem that happens to be invisible; it is a whole-document ingest artifact from one batch of source files, and the court field is simply where it becomes load-bearing. Anyone extracting entities or matching phrases across the text of that batch is working with the marks in place.

Third result: the seam you can see without a hex editor

The invisible marks are the eye-catching finding. They are not the expensive one. We pulled our own court facet — the top twenty courts for a query, the kind of table a buyer skims in an evaluation — and found the same court listed twice, in plain visible characters, in the first five rows.

A Chinese intermediate court has two written styles. The full official style names the province and then the city; the common style drops the province. Both are correct; neither is a typo. In one province’s labour dispute facet, both were present:

The published table’s second row is the same court as its first row. The court that genuinely ranks second is published fifth, carrying 40 per cent of its own record count. Anyone who builds a most-active-courts ranking from this table gets the order wrong and every number understated, and nothing in the table hints at it.

We then ran the same collision test — strip a leading province name from every court value in the facet, count the names that collide — across twelve slices:

Thirty distinct courts turned up split across those twelve slices. The two zeroes are informative rather than reassuring. Beijing returns zero because it is a municipality: the province-level name is part of every one of its courts’ canonical names, so there is no short form to diverge into. The private lending slice returns zero because its top twenty happens to contain no intermediate courts, and the collision is almost entirely an intermediate-court phenomenon — basic-level courts are conventionally named for a city and district, where the province prefix is rare in both styles.

Note also what the query does to this number. The same province measured with a search term shows 44.1 per cent of the table sitting in split buckets and 8.3 per cent with no query at all. As with the dirt rate, the visible severity of this defect is a property of what you asked for. It is not a corpus constant and we are not going to publish it as one.

Proving the two names are one court

A name collision is suggestive, not conclusive — so we checked it against a field that does not depend on the name at all. Chinese case numbers embed a court code. Pulling records from both spellings across five queries and two provinces:

Same court code, both spellings. And the two spellings are almost perfectly disjoint in time and completely disjoint by source. Across every intermediate court in that pull, 184 of the 187 short-style records came from the oldest dataset family, and not one full-style record did.

That is the whole mechanism. The naming style is a property of which ingest generation a record arrived in, and the ingest generations partition the corpus roughly by era. Group by court name and Guangzhou’s intermediate court stops hearing cases after 2017; a court with a longer name opens in 2020 and hears the rest. Neither curve is the court’s caseload. The gap between them is a merge seam.

We want to be precise about what this is and is not. It is not corrupted data: every one of those records carries a correct name for its court, in a style that court genuinely uses. It is not something a schema check catches, because there is no schema violation. It is an entity resolution failure, and entity resolution is exactly the layer that a corpus assembled from multiple source generations needs and that record-level quality flags cannot supply.

What this does to yesterday’s number

Yesterday we reported that 53.9 per cent of our read-time-derived provinces could not be checked against anything else in the record, because the court name was present but named only a city or district, or was missing entirely. Today’s sample, different queries and a day later, puts the same figure at 67.0 per cent of 106 derived records. Two numbers fourteen points apart for the same quantity is itself the result: the unfalsifiable share is not a corpus property, and anyone quoting either figure as one is over-reading it, ourselves included.

Today’s work says something about why it moves. Whether a court name states a province at all depends heavily on which ingest generation the record came from:

A record’s auditability, in other words, is largely inherited from the batch it arrived in rather than from anything about the case. Yesterday’s 53.9 per cent and today’s 67.0 per cent are both partly measuring the dataset mix that our query happened to pull. We did not know that yesterday and we published the number without the caveat it needed.

Two smaller items reproduce from yesterday on today’s independent sample. Thirty records — 0.85 per cent of the sample — have a derived province and no usable court name, so nothing in the record can contradict either field. And the reverse defect is still there: four records whose upstream province is empty and which our derivation declined to fill, while the court field in the same record names the province outright. The derivation reads one field. The answer is in the next one.

Five checks, all of which cost minutes

1. Strip the leading province name from every court value in any facet you are shown, and count collisions. One line of code, no special access. Seven of our eight query slices failed it.
2. Count the court values whose first character is not in the script you expect. This is the only cheap test that surfaces invisible leading characters, and it is what found ours.
3. Ask what a validity flag actually inspects, not whether it is accurate. Ours is accurate and checks the end of a string while the damage sits at the start.
4. Run your audit twice, once by browsing and once by searching. If the two disagree, the browse number is the one that is lying to you about what your product will see.
5. Cross-tabulate any per-record quality metric against the source batch. If the metric tracks the batch more than the content, you are measuring provenance and calling it quality.

What we did not do

We did not fix any of this today. Normalising court names is a change to the index, not to a document page, and merging two spellings is a decision with consequences for anyone already joining on the current values; it belongs in a change log with notice, not in a quiet edit. We have not tested any other vendor’s corpus and make no comparison to one. We did not measure how many courts are split corpus-wide, only how many surfaced in twelve top-twenty tables. The safety gate that declines to guess a province when a regional character is ambiguous fired zero times in 3,550 records today, as it did in yesterday’s 2,522; a guard clause that has never fired is untested, not proven safe. And every rate here belongs to an ordered page walk on one day.

The reason we publish this rather than fix it quietly first is that the class of defect generalises past us. Any corpus assembled from more than one source generation has these seams, most of them are invisible at the record level, and a validity flag can be perfectly accurate about a question nobody is asking. If you are evaluating Chinese case law data from anyone, run check one on their facet before you run anything else.