The De-duplication Rule That Deletes the Field You Wanted

Yesterday we published a finding and left a hole in it. We had shown that the same judgment can sit in our corpus several times over, and that the copies disagree with each other about the case — so collapsing them is a metadata decision, not a row-count decision. What we did not give was a rule for deciding which copy lives. Without that, the finding is an observation a buyer cannot act on.

Today we implemented the rule almost everyone reaches for first — keep the copy with the most populated fields — and measured what it does to our own data. The short answer: on one date it is excellent, on another it is destructive, and the difference is not in the rule at all. Along the way the measurement forced us to correct one of yesterday’s own published numbers.

The rule, and the first thing wrong with it

The rule is simple enough to write in a sentence. Group records that share a case number; score each copy by how many of six metadata fields are non-empty — court, cause of action, case type, procedure, province, judgment year; keep the highest scorer. It needs no reference data and no human. That is exactly why it is the default.

Before asking whether its answers are good, ask how often it produces one at all. A rule that ties has not decided anything; it has quietly handed the choice back to whatever order your rows happened to arrive in.

On the most recent date the rule settles two cases out of 196. Everywhere else the copies are equally populated and the tie-break falls through to insertion order — which is to say, to nothing. A rule that is silent 99 per cent of the time is not a policy; it is a coin flip with a policy’s reputation. Hold that date; it turns out to be the interesting one for a different reason.

Same rule, opposite outcomes

Now the quality question. For each repeated case number we compared three strategies on the field buyers filter on most, the cause of action: the expected coverage if you keep an arbitrary copy, the coverage delivered by the most-populated-copy rule, and the coverage available under a field-wise merge that takes any non-empty value from any copy.

On 27 September 2021 the rule is a large win: it nearly doubles cause-of-action coverage and leaves nothing on the table. On 12 June 2019 the same rule, unchanged, performs worse than picking a copy at random — it drops coverage by 23 percentage points against the arbitrary baseline and forfeits 52.8 percentage points against what was actually available. Concretely: on that date the rule’s chosen survivor has an empty cause of action although another copy in the same group carried one in 1,504 of 2,849 repeated case numbers, 52.8 per cent.

So the useful question is not “is this a good rule”. It is what makes it good on one date and destructive on another.

The answer: batches are complementary, and counting cannot see it

Yesterday we established that in a merged corpus, field completeness is close to binary at the batch level. Put two such batches on the same case and the consequence follows immediately. Here is 12 June 2019 broken out by the batch that supplied each document.

The dominant batch scores four or five populated fields on almost every record and has no cause of action at all. Small batch A scores two and is the only place the cause of action exists. Counting fields therefore hands the dominant batch a permanent, structural victory, and the surviving record is stripped of the single attribute that made the duplicate worth having. The attribution is unambiguous: of the 1,504 groups that lost a cause of action, 1,451 were the dominant batch beating small batch A.

Compare 27 September 2021, where the rule performed beautifully. There the overlapping batches were not complementary but nested — one simply carried more of everything — so “most populated” and “most correct” pointed the same way. The rule never changed. The batch pair did. Any de-duplication policy quoted to you without the batch pairing it was measured on is a policy whose behaviour is unknown.

Populated is not correct, and the rule rewards populated

There is a worse failure mode hiding inside “most populated”: a field that is filled in with the wrong value scores exactly like a field that is filled in with the right one.

PRC case numbers are useful here, because they carry their own validator. We established the mechanics separately — the case number is the only honest province field we hold, since the number encodes the issuing court including a province character. What is new today is using it as a de-duplication instrument: the court and province fields on a record can be checked against the case number on that same record, with no external reference data and no judgement call, which turns a class of apparent conflicts between copies into a verdict about one of them. We ran that check across every batch appearing on the four dates.

The result was flat — every batch contradicted its own case numbers on 0.0 to 0.1 per cent of testable records — with a single exception. In one batch, 33.3 per cent of the records whose court field names a province contradicted the province encoded in the case number. Widening from that date to the whole batch, about 4.4 million records, the shape of the corruption is plain:

A single trial-level docket cannot have been decided by the Supreme People’s Court, and one provincial high court cannot account for a twelfth of a nationwide batch. Two court values are absorbing a quarter of that batch, and the case numbers on those records name district courts in other provinces. This is not sparse data. It is dense, confident, wrong data, and a field-count rule reads it as a point in that copy’s favour.

How often does that actually bite? On 12 June 2019, 779 of 2,849 repeated case numbers (27.3 per cent) contain at least one copy whose court contradicts its own case number. The rule selected that contradicting copy in 70 of them, and in 64 of those a clean copy was sitting in the same group, unselected. Small in absolute terms — and entirely invisible to any check that only counts populated fields.

The correction: some of what we called duplicates were not duplicates

The most recent date, where the rule tied 99 per cent of the time, tied for a reason we had not considered. We inspected the groups instead of counting them.

They are not copies. They are different instruments sharing one case number: a judgment and a ruling on the same docket; an enforcement ruling and an enforcement notice; a decision and a ruling. The copies tie on field count because both are complete records — of different documents. Collapsing them would not have removed redundancy; it would have deleted a document that exists.

This corrects a number we published yesterday. We reported redundancy of 0.61 per cent on 11 April 2023 as a duplicate rate. Under a key that includes the document type it is 0.02 per cent, because 96.4 per cent of those groups were judgment-and-ruling pairs rather than repeats. The other three dates move by fifteen basis points or less, so yesterday’s substantive finding stands; the most recent date’s figure does not, and we have annotated that article accordingly. The general lesson is the one that costs money: a de-duplication key that does not include the document type will silently destroy real instruments, and it will do so most aggressively on the newest data, where each docket has more of its instruments published.

There is a cheap tell for distinguishing the two cases, and it is worth stating because it costs one comparison per group. Within repeated case numbers, the share whose titles are identical ran 50.1 per cent on 18 May 2016, 43.0 per cent on 12 June 2019, 98.1 per cent on 27 September 2021 and 0.0 per cent on 11 April 2023. The date where the survivorship rule was most useful is the date where the copies are near-identical documents; the date where it was useless is the date where they were never copies at all. A group whose titles all differ deserves inspection before it is collapsed, and titles are one field away in any corpus that has them.

What we would actually ship

Putting the measurements together gives a protocol rather than a rule. Each step below exists because one of the failures above forced it.

1. Key on case number plus document type. Case number alone over-groups; on one of our four dates it over-grouped 96.4 per cent of the groups it found.
2. Merge field-wise; never keep one document and discard the rest. Document-level survivorship cost us 52.8 percentage points of cause-of-action coverage on a single date. Field-wise merge recovered it: 91.9 per cent against the rule’s 39.1 per cent.
3. Assign trust per batch and per field, not per record. Population and correctness are both batch properties. The batch that supplies the cause of action on that date is the same batch whose court field is unreliable — so it should be trusted for one field and distrusted for another. No record-level score can express that.
4. Run an internal validator before you trust a populated value. The court code inside the case number needs no reference data. On 12 June 2019 it reclassified 684 of 963 apparent court conflicts from “two sources disagree” to “one source is wrong” — which is a repair, not an arbitration.
5. Publish the residual arbitration budget. After field-wise merge and the validator, what remains is the set of conflicts a machine genuinely cannot settle. That number is the honest measure of how much human work a corpus still owes.

Ours, on the four dates, after steps one to four:

Two things are worth reading off that table. The residual is small — between 3 and 59 groups per 10,000 documents, so the arbitration problem is bounded and fundable rather than open-ended. And it is concentrated: most of what looked like disagreement in the raw data was vocabulary rather than contradiction — one batch storing a procedure as a numeric code where another stores a word, one storing the document type where another stores the case type, one writing a province suffix another omits. Those need a crosswalk, written once. What survives a crosswalk and a validator is the real work, and it is a few dozen groups per ten thousand documents.

What we are not claiming

We measured no other provider, so nothing here ranks anyone against anyone. The four dates were chosen to span ingest eras, which makes them worked examples and not an annual rate — the residual budget in particular will move with the batch mix of whatever slice you care about. We have not applied any of this: normalising a vocabulary or choosing a surviving value changes results for anyone who has already joined against current values, so it belongs in a change log with notice, not in a quiet overwrite. The corrupted court field in that one batch is now a known defect on our list rather than a fixed one. And the validator itself only covers court and province, because those are the fields the case number encodes; a cause of action cannot be checked this way, which is precisely why the cause of action is what remains in the residual column on two of the four dates.

If you are evaluating any Chinese case law corpus, including ours, the four questions that follow from all of this are cheap to ask and hard to answer vaguely: what is your de-duplication key; what happens to a field the surviving copy does not have; how do you distinguish a populated field from a correct one; and what is your residual arbitration count. We have now answered all four on our own data, in public, with the failures included.