We Re-Ran Our Own Accuracy Checks With a Better Sampler: One Figure Was Right for the Wrong Reason, and 289 Phrase Matches Had No Phrase

Yesterday we published a defect in our own sampler: a seeded random_score on _seq_no is shard-local, so a “random” 1,500-record sample came from two of our fifteen ingest batches and reported zero hits for a defect present in 19.3 million records. We then did the honest but lazy thing and hung a blanket note on the articles whose precision figures came from samples, saying they might be affected.

Today we went back and re-measured all of them. Four of the five published precision claims survive; one of them survives for a reason we had wrong; the blanket note was itself too broad and is corrected below. And re-running the checks turned up a defect we had not looked for: the same analyzer property that makes a Chinese phrase search return a false zero also makes it return false positives — 289 records that match the phrase 出资加速到期 and do not contain it.

Everything below was measured on 19 September 2026 against our index of 160,291,678 records in fifteen ingest batches. Every sample in this post records its seed, its size and the batches it drew from, so the numbers can be reproduced rather than trusted.

What a stratified draw actually looks like

The fix for a shard-local seed is not a better seed. It is to stop asking the coordinating node for a global top-k at all. For every check below we ran one query per ingest batch, allocated the sample across batches in proportion to each batch’s population, and reported the batch spread alongside the result:

for ds in datasets:                    # from a terms agg on dataset
    n_ds = round(N * pop[ds] / total)  # proportional allocation
    search(size = n_ds,
           filter = [query, {"term": {"dataset": ds}}],
           random_score = {"seed": 20260919, "field": "_seq_no"})

Within a single batch the shard tie-break still exists, but a batch is far more homogeneous than the corpus, so the damage is bounded and, more importantly, visible: if a batch is missing from the output you can see it, because you asked for it by name.

The claim classifier: right about the rule, wrong about why

On 17 September we reported precision for a rule that sorts equity transfer plaintiffs into claim groups by reading the passage after 诉讼请求. The published figures were 20 of 20 for unwinding claims, 17 of 20 for price claims and 11 of 20 for confirm-or-register claims. We redrew each group at n=40 with seed 20260919 and read all 120 passages.

One thing the larger sample makes visible that n=20 did not. The unwinding group is 40 of 40 against its own stated definition — every record asked for rescission, invalidity, revocation or return — but in 8 of the 40 what the plaintiff wants returned is investment money rather than the consideration for a share transfer: “return the partnership funds”, “refund the subscription for original shares”, “return the deposit paid under the letter of intent”. The rule is doing exactly what it says; the label “unwinding claim” is looser than a lawyer would use it. That is a definitional caveat, not a defect, and a precision figure cannot surface it — only reading the sample can.

The five price-claim misses split the way the original described them: three are passages where the rule looked in the wrong place — it landed in the court’s reasoning or the defendant’s answer rather than the claim list — and two are genuine misclassifications, one a claim for a share of profits and one a claim to confirm a transfer and register it.

The confirm-or-register group is where the interesting thing happened. Read on its own, 30 of 40 looks like a large improvement on the published 11 of 20. It is not an improvement. Sixteen of the 40 records in the new sample are the China Shanshui mass-filing cluster — the two courts we wrote about on 17 September, which between them supply 2,502 of the 5,542 judgments in this group. Those sixteen have identical, textbook-clean claim passages, because they are the same pleading filed 3,219 times. Split the sample:

The published 55 per cent was right, and the 75 per cent is an artefact. A precision figure for a category that contains a mass-filing cluster is partly a measurement of the cluster, and the bigger the cluster the better the rule will appear to perform on exactly the records nobody needs a rule for. This is a general hazard, not one about our rule: any accuracy number you are quoted for a Chinese case-law category should come with the share of that category held by its largest filer.

Correcting our own correction

The note we added on 18 September said the 17 September precision figures may have been drawn from a stratification-biased sample. They were not. That article’s sample was drawn from a local scroll of the entire category — 215,830 de-duplicated documents spanning fourteen of the fifteen batches — not from a top-k query, so the shard tie-break never applied to it. The real defect in that sample was different and smaller: the seed was not recorded, so the figures were not reproducible. They are now.

Six phrase checks, re-run across every batch

On 14 September we reported that four Chinese phrases return zero match_phrase hits in our index under the default query analyzer and thousands under the index-time analyzer, and that 60 of 60 sampled records in each recovered set contain the literal string. Those samples were drawn from Elasticsearch, so they are exactly the kind the shard tie-break can spoil. Re-run with proportional allocation across every batch that holds hits:

382 of 382, every batch with hits represented. The published claim holds. One detail the stratified view adds for free: Article 57 lives in six batches and 308 of its 390 records sit in one of them, which is what you would expect of a provision in force since 1 July 2024 and nothing more.

The defect we were not looking for: a phrase match is not a literal match

The check above asks whether a record returned by a phrase query contains the phrase. We had only ever run it on recovered false zeros. Run it on a phrase that works normally and the answer changes.

Scanning all 16,398 records that match_phrase returns for 出资加速到期 (acceleration of capital contributions), 289 — 1.76 per cent — do not contain that string anywhere in the body. They are not errors of retrieval in the ordinary sense. They are texts where something sits between the two halves of the phrase and is discarded before the index sees it:

Run the analyzer on that third form and the mechanism is visible: 认缴出资“加速到期” produces the tokens 认缴, 出资, 加速, 到期 at positions 0, 1, 2 and 3. The punctuation is dropped and the positions close up, so a phrase query with zero slop lines up against text that does not contain the phrase.

Two consequences, and they point in opposite directions from the same cause. A phrase count in Chinese is not a count of the literal string: it is a count of token adjacency after punctuation is removed. For a doctrinal search that is usually the behaviour you want, because 出资“加速到期” means what 出资加速到期 means — all 289 of these records are on the doctrine. For anything that hands a customer a phrase count as evidence of a literal string — a citation check, a plagiarism screen, a clause-presence test in a retrieval evaluation — it is a silent false positive rate of about 2 per cent on one phrase in one category. We did not measure it corpus-wide. The same scan over the second phrase family, 提前缴纳出资, found 1 of 2,425.

A seed is not enough either

The 13 September acceleration piece did record its seed and did use random_score, which we had treated as the careful version. Redrawing the three phrase families stratified shows what the careful version still missed:

Group B is the one that moves, and the reason is a restriction we disclosed but did not weight. The published sample was drawn from records dated on or after the revised Company Law came into force. 69.6 per cent of group B is dated on or after that date; 30.4 per cent is not, and the two misses in the full-range sample both come from the older material, where 提前缴纳出资 is contract language about a shareholder who agreed to pay in early rather than a creditor invoking the rule. A reader who took 25 of 25 as the precision of group B as counted was reading a figure for 70 per cent of it. The precision of the group as counted is 31 of 33, and we would not report either number to three significant figures at these sample sizes.

Group A is the cleaner lesson. A seeded sample of 25 drew nothing from the 2024 batch, which holds 21 per cent of the group — not because of anything in the data, but because of how ties are broken during the merge. On this occasion it cost nothing: the stratified redraw is 34 of 34. On 18 September the same mechanism cost us a defect in 19.3 million records.

Two older year tables, re-checked against the date field

Three days ago we published that the date field in three of our fifteen batches is closer to a publication date than to the date a judgment was signed. That put two earlier articles with year tables on the list to re-check. Both have now been re-run.

The batch-provenance piece survives. Its central table counts labour-dispute records per 10,000 documents in each judgment year, once through the cause field and once through a text control. Excluding the three affected batches entirely moves nothing outside the noise:

The eighty-four per cent fall from 2015 to 2020 and the ninefold recovery to 2023 are unchanged. That table happens to omit a 2024 row, which is the only row the defect could have wrecked.

The case-type piece needs a note. That article reported a clean edge inside the 2024 row: the registry-class case-type vocabulary’s last dated record is 31 October 2024, and the short vocabulary takes over from 1 November, with only 41 short-vocabulary records dated earlier. Every one of those numbers reproduces exactly. But 99.999 per cent of the 2024 row comes from a single ingest batch, and that batch is one of the three whose date is a publication date. We sampled 400 records on each side of the edge and parsed the signature date from the end of the text:

So the edge is real and it is sharp, but it is a boundary in the publisher’s pipeline, not a date on which Chinese courts changed how they classify cases. It also means the two sides are not comparable: before it, half the records carry a date up to three years after they were signed; after it, the date is the signature date. That strengthens the article’s thesis — the case-type field is stratified by ingest, not by law — and it removes a reading the article left open. A note to that effect goes on the page today.

What we did not re-check, and why

The fourth article on the list, on an 80,000-row structured labour dataset, reports completeness and literal-recoverability figures that are full counts over the shipped file, not samples: 70,564 of 80,000 rows populated, 40,931 of 80,000 excerpts containing the shipped compensation figure. There is no sampling method in them to correct. The one sampled claim in that piece — 20 rows hand-read out of 865 flagged, of which 18 were polarity errors — was drawn from the local file rather than from the index, so the shard tie-break does not reach it. The article already says that 90 per cent carries a wide interval at n=20, and it still does.

How to run this on a vendor

• Ask for the seed, the sample size and the batch composition of every accuracy figure you are quoted. A figure without the third is not reproducible even if the first two are present.
• Ask what share of the category comes from its largest single filer. If a cluster is 40 per cent of a category, a precision figure for that category is mostly a statement about one pleading.
• For any Chinese phrase count offered as evidence that a string appears in a document, ask what fraction of the hits contain the literal string. The answer is not 100 per cent, and the shortfall is systematic rather than random.
• For any table broken down by year, ask which ingest batches supply each row and whether the date in those batches is the date the judgment was signed.

What we are not claiming

These are our own re-reads of our own data, not independent replication. Sample sizes of 33 to 40 per group give intervals wide enough that most of the movements above are not statistically distinguishable from the published figures; the claim is that the figures do not move materially once the sampler is fixed, not that the new figures are precise. The 1.76 per cent false-positive rate is measured on one phrase within one category and is not a corpus-wide rate. We read claim passages and doctrinal passages, not case outcomes, so nothing here revises any rate we have published. And none of the underlying defects is fixed: the shard-local seed is still how random_score behaves, the punctuation is still dropped at analysis, the three batches still carry publication dates, and the 19.3 million rulings reported yesterday still open by calling themselves judgments.