Evaluating a Chinese Case Law Vendor: A Coverage Verification Protocol You Can Run Before You Sign

Every supplier of Chinese court data, ourselves included, leads with a number. The number is not the thing you are buying. What you are buying is whether the matters your product needs are present, complete enough to use, arriving fast enough to stay current, and identified consistently enough to join to anything else.

None of that is visible from a slide, and none of it is settled by asking the supplier a harder version of the same question. It is settled by running a short protocol yourself, against a trial key, on a workload you define. This piece sets out five tests that fit inside a week, the five artefacts worth asking for in writing, and — at the end — the ways this protocol can still mislead you.

Why headline counts cannot be compared

Start by discarding the comparison you were probably asked to make. Three things break it, and all three can be true of numbers that are individually honest.

The unit is not standard. A document, a case, a dispute and a record are four different things. One dispute regularly produces a first-instance judgment, an appellate judgment, possibly a retrial decision and several enforcement rulings. Counting documents and counting disputes can differ by a large factor without anyone misrepresenting anything, and the ratio is not constant across case types.

There is no denominator. The set of published PRC judgments is itself a product of publication practice, which has varied across years, courts and case categories. Because nobody can state the true size of the universe, nobody can state a defensible percentage of it. Any supplier claiming a coverage share is quoting a ratio whose denominator they invented — including if that supplier is us.

A count has no timestamp. Corpus size is a snapshot. It tells you nothing about the rate at which new decisions arrive, which is the variable that actually decays under you after signature. A corpus that is larger today and refreshed quarterly loses to a smaller one refreshed weekly for most production use.

The practical consequence: stop trying to rank suppliers on size and start measuring each one against your own workload. The rest of this page is how.

The protocol at a glance

Tests 1 and 4 use facts you already hold, so they carry the most evidential weight. Tests 3 and 5 need calendar time, which is the main argument for starting the trial before procurement rather than during it.

Test 1: known-item recall against a seed list you own

Build a list of matters you already know exist — from your own client work, from decisions cited in secondary literature, from a prior vendor's exports, from anything where you hold the citation independently. Aim for scale in the low hundreds. Then query for each and score three outcomes rather than two: found, missing, and found in the wrong version (an earlier instance returned for an appellate citation, or a procedural ruling returned for a judgment).

The third bucket is the one that matters and the one a naive script collapses into the first. A pipeline that reports high recall while quietly returning the first-instance decision whenever the appellate one is missing will look excellent in evaluation and fail in production, where the answer depends on which court spoke last.

Stratification decides whether this test is worth running at all. A seed list drawn from recent, prominent, commercially significant decisions will make every supplier look strong, because that is the easiest slice for all of them. Spread the list deliberately across decades, court levels, regions and case types, and keep the per-stratum counts visible when you score, because the useful output is not a single recall figure but a map of where the misses cluster. Misses concentrated in one era or one court level are a coverage boundary you can plan around, and possibly negotiate. Misses scattered evenly are a retrieval or normalisation problem, which is a different conversation and often a fixable one.

Split the list. Send half to the supplier — a scoped sample is what lets them answer precisely, and per the pricing mechanics, buyers who arrive with a defined scope get faster and usually better quotes. Hold the other half back and run it yourself afterwards.

Test 2: field completeness within strata, never overall

Ask for population rates per field, crossed with year band and court level. An overall completeness percentage is a weighted average dominated by whichever period holds the most volume, so a field can read as well populated while being nearly absent from everything before a certain year.

The fields worth insisting on, roughly in the order that their absence hurts:

• Case number, normalised — everything else joins through it
• Issuing court and level, as a code rather than a name string
• Decision date, distinguished from any publication date
• Cause of action / case category, and which taxonomy version it follows
• Procedural posture — instance, and whether the document is a judgment, ruling or enforcement decision
• Disposition, and whether it is extracted as a field or must be read out of the text
• Reasoning section boundaries, which decide whether downstream work is configuration or an NLP project (see corpus preparation)

Two answers should change your posture. If the matrix arrives quickly, the supplier measures their own data — a stronger signal than any claim in the matrix itself. If the answer comes back as an adjective, you have learned that the instrumentation does not exist, which is worth knowing before it becomes your problem.

Test 3: freshness as a distribution, not a best case

Freshness is usually quoted as a single interval. Measure the shape instead. Choose a decision week that has already passed, count how many records with decision dates in that week are available now, then repeat the count a month later against the same week. The gap between the two measurements is the tail.

Two things to define before you start, because otherwise the numbers are not comparable to anything. First, the reference date: lag from the decision date, from the publication date at source, and from arrival in the supplier's system are three different quantities, and only the first is meaningful to your users. Second, the stratum: publication timing varies by case type and court level, so an average across everything can look healthy while the category your product depends on runs materially behind it.

This is also the test whose result you should expect to be imperfect everywhere. Suppliers do not control when courts publish. What separates them is whether they measure the lag, disclose its shape, and tell you when it changes.

Test 4: identity keys and appeal linkage

Take roughly twenty disputes where you independently know an appeal exists. For each, check whether the appellate document is present, and then check the harder thing: whether the relationship is in the data, or whether you would have to reconstruct it yourself by matching party names and case numbers with a regular expression.

The cost of getting this wrong is not retrieval quality. It is that instance chains are the unit at which a legal corpus leaks. If linkage is absent, every evaluation split you build later is a random split over documents that share facts, parties and long passages of recited text — and the score you report will not reproduce in production. That mechanism is set out in full in the corpus preparation piece; here the point is narrower: this is a property to test at trial, because it is very expensive to add afterwards and it is not visible in a sample of individual documents.

While you are there, ask what constitutes the identity key and how case numbers are normalised. Formatting conventions vary across decades and courts, and a supplier who cannot state the key precisely is telling you that deduplication and joins are your problem now. The field-level shape of a well-formed record is covered in the API structure walkthrough and structuring PRC court data for AI.

Test 5: stability across two deliveries

Run one identical, scoped query — a fixed case type, court level and year range — at the start of the trial and again a month later. Store both result sets and a content hash. Then ask the supplier to account for the difference.

Counts moving upward because courts published more decisions in that window is expected and healthy. Counts moving because an extraction pass was re-run over historical data is a different event entirely: it means the corpus underneath you is mutable, and if you have already trained or indexed against it, your artefacts no longer correspond to anything the supplier can reproduce. Neither answer disqualifies anyone. Not knowing which one happened does.

The five artefacts to ask for in writing

Each should arrive as a table or a specification, not prose. Requesting them before a contract exists is normal, and how fast they come back is itself the measurement.

1  coverage distribution   records by year x court level x region x case type
2  completeness matrix     field population rates, crossed with the same strata
3  freshness method        how lag is measured, from which reference date
4  identity key spec       exact fields forming the key; case-number normalisation
5  change log              re-extractions, backfills, removals — each with a date

Item 5 is the one buyers forget and the one that determines whether tests 3 and 5 stay meaningful after signature. A supplier who publishes change events lets you distinguish "the world changed" from "our pipeline changed" for as long as the relationship lasts. A supplier who does not leaves you re-running this protocol every quarter, informally, to no standard.

Four ways this protocol can mislead you

1. Small samples have wide error bars. A few hundred seed items across many strata leaves very few per cell. Treat per-stratum results as directional, and resist converting them into a percentage that will get quoted internally as if it were measured on the whole corpus.
2. Your ground truth is biased toward your own practice. That is a feature for a fit assessment and a defect for a general one. A supplier who scores poorly on your seed list may be strong in areas you did not test, which matters if your product roadmap moves.
3. The half you send can be prepared for. This is exactly why the list is split, and why tests 2, 3 and 5 exist: a completeness matrix crossed with era, a lag window you choose, and a comparison across two deliveries are all harder to special-case than a list of citations.
4. Passing says nothing about the commercial terms. Coverage and quality are separable from what you are permitted to do with the data. Granted uses, survival after termination and refresh obligations are decided in the agreement, not in the trial — the term-sheet piece covers those in the order they cost money.

One more caveat that applies to this page specifically: we describe a corpus of more than 160 million publicly available PRC court judgment records, and field completeness across it varies with decades of source formatting. That is precisely why the answer to test 2 has to be a distribution table rather than an adjective — including when the supplier being tested is us. Run the protocol against us. If a cell comes back thin, we would rather you find it in week one than in month nine.

For the decisions upstream and downstream of this one: license versus scrape covers build-versus-buy, the source portal's robots.txt read line by line is worth checking before anyone budgets a crawler, the academic-benchmark comparison explains why a research dataset does not carry a product, the licensing guide and the delivery-model comparison cover the commercial shape, and the 2026 market overview sets the landscape. On the access side, see the English-language API and the MCP server; on the serving side, the retrieval pipeline walkthrough.