From Licensed Corpus to Training Run: Preparing Chinese Court Judgments for Fine-tuning and Evaluation

The contract is signed, the first delivery has landed, and a folder of PRC court judgments is sitting in object storage. This is the moment where most write-ups on Chinese legal data stop, and where the engineering work actually starts.

A licensed corpus is not a training set. Between the two sits a series of decisions — what counts as a duplicate, which text to strip, how to split, what to record — that determine whether the numbers you eventually report mean anything. They also tend to get made in an afternoon, by whoever is closest to the pipeline, and are then hard to revisit when a customer asks how the evaluation set was constructed.

This piece works through the five that matter most for Chinese judgment data specifically. It is written for the person who will run the job rather than the person who signed the agreement, though the two need to have read the same page at least once.

First: which of the three jobs is this corpus doing?

"Training data" is three different requirements wearing one label, and a corpus prepared well for one is often prepared badly for another. Fix this before touching the files, because the deduplication threshold, the stripping rules and the split strategy all follow from it.

The third row is the one teams under-resource. An evaluation set is the only artefact here whose entire value comes from what it excludes, and it is usually built last, from whatever is left over, by a random split.

Deduplication: four problems, not one

Duplication in a PRC judgment corpus is structural rather than accidental, and a single cosine-similarity threshold applied across the whole set will make things worse in both directions — leaving genuine redundancy in place while deleting documents that were never duplicates at all. Separate the four cases.

The expensive mistake is collapsing rows 2 and 3 into one rule. A pipeline aggressive enough to thin out template-driven filings will also, on any text-similarity measure, treat an appellate judgment as a near-duplicate of the first-instance decision it reviews — because it recites it at length before disagreeing with it. What gets deleted is precisely the passage where a higher court explains why the reasoning below was wrong, which is the text a legal model most needs to see.

So: do identity-based deduplication (rows 1 and 4) first and cheaply, since it reduces everything downstream. Do similarity-based work (row 2) last, scoped to a case category, with the chain structure from row 3 already attached so a chain member can never be dropped in isolation.

Boilerplate: strip by section, never by regex over the whole document

PRC judgments are highly structured documents. Procedural recitals, service-of-process language, standard notices about appeal windows and formulaic closing sections repeat at enormous scale and carry little task-relevant information. The court's reasoning section is where the decision is actually explained — and for anything involving outcome prediction or explanation, it is the part worth targeting.

That argues for stripping. It does not argue for stripping with a pattern match across raw text: formatting conventions vary across decades and court levels, so a pattern tuned on recent documents will silently mangle older ones, and the same phrase can be scaffolding in one section and substantive in another.

The approach that survives review is boring: segment first into labelled sections, make the strip decision per section per job, and keep the original document addressable so any decision can be reversed without asking the licensor for a fresh delivery. If your delivery arrives with section boundaries already identified, this becomes a configuration choice rather than an NLP project — a large part of why field depth is worth negotiating up front. We walk through those fields in the API structure piece and in structuring PRC court data for AI.

Coverage skew is a property to document, not a bug to sample away

The distribution of published PRC judgments across years, case types, court levels and regions is uneven, and the unevenness reflects publication practice as much as it reflects underlying litigation. Not every matter produces a public document: cases resolved by mediation, withdrawal or settlement often leave no published judgment, and arbitration sits outside the public court record entirely.

Teams meet this fact and reach for resampling. It is the wrong instinct: reweighting a corpus to look balanced requires a target distribution you cannot verify, and it converts a measurable property of the data into an assumption buried in a preprocessing script. What to do instead, in order:

1. Measure the distribution along year, court level, region and case type before any filtering, and store the table.
2. Stratify deliberately for the task. If the product answers construction-dispute questions, sampling that category heavily is sound engineering — state it as a design choice rather than performing it silently.
3. Re-measure after every filtering stage. Deduplication and boilerplate rules are not distribution-neutral; template-heavy categories lose the most volume, which shifts the mix in ways nobody intended.
4. Ship the distribution with the model. The team fielding the customer question about representativeness is not the team that built the pipeline.

The honest framing for anything you publish downstream: a corpus of published judgments supports statements about what published judgments contain. Turning that into a claim about litigation outcomes in general requires an assumption about the unpublished remainder that the data cannot supply. That constraint applies to us too, and it belongs in the model card beside the coverage table.

Leakage: the unit that repeats is the dispute, not the document

This is where otherwise careful teams report numbers that later collapse. A random train–test split assumes documents are independent draws. In case law they are not, and PRC judgments give you at least three ways for the assumption to fail:

• Instance chains. One dispute produces first-instance, appellate, retrial and enforcement documents that share parties, facts and often verbatim recited passages. Randomly split, the test set contains matters the model has effectively already read.
• Serial litigation. The same parties appear across many related matters, and a model can recognise the relationship rather than reason about the law.
• Batch filings. Mass-filed matters sharing a single template distribute themselves across both sides of a random boundary, inflating scores for template-shaped tasks specifically.

The fix is not exotic. Split on a dispute-level identity key rather than a document key; keep whole chains on one side of the boundary; and add a time-based holdout so at least one evaluation slice consists only of matters decided after everything in training. The time split is the one that predicts production behaviour, because production is always asking about matters the model has not seen.

Two smaller habits pay for themselves. Record the excluded slices in the manifest, so a later retraining run does not quietly absorb your evaluation set. And when reporting, state the split rule alongside the score — a retrieval metric without a split rule is not a comparable number, and increasingly the reviewer knows it.

The evidence trail: what procurement will eventually ask for

The party who asks how your training data was assembled is usually not the licensor. It is your enterprise customer's procurement or security review, arriving between the pilot and the contract, with a questionnaire. The difference between a one-week and a one-quarter review is whether the answers already exist in a file. So keep a manifest per shard, generated by the pipeline rather than written afterwards:

source          supplier, delivery ID, delivery date
grant           agreement reference + the specific granted-use clauses relied on
scope           case types, year range, court levels, regions requested
counts          records in, records out, per processing stage
completeness    field-population profile at delivery
transforms      every filter, threshold and rule applied, with version
splits          which slices are held out and why
exclusions      anything removed on request or on legal review, with date
integrity       content hash of the shard as trained on

Nine lines, and the fifth and sixth are the ones people skip and later regret. A completeness profile taken at delivery is also your only defence against a slow drift where a supplier's extraction changes and nothing in your metrics notices for two quarters.

One item sits across the boundary between engineering and contract. The artefacts you are about to create are the things the licence has to cover — embedding indexes, derived fields, evaluation sets, model weights. Pre-training, fine-tuning, evaluation, embedding generation, retrieval indexing and display of extracts are separate permissions, and silence is not permission. Confirm which clause covers each artefact before the first run and record it in the manifest; the term-sheet piece sets out the clauses in the order they cost you money.

Five red flags in a corpus you have just received

• No delivery-time counts. If nobody recorded how many records arrived, every downstream count is unanchored and no processing stage can be audited.
• Field completeness described with an adjective. "Rich metadata" is not a profile. Ask for population rates by field and by era; formatting conventions changed over the decades the corpus spans.
• No case-number normalisation. Without a stable identity key you cannot deduplicate reliably, cannot build chains, and therefore cannot build a leakage-safe split.
• An evaluation set built last, by random split. The most common reason a legal AI benchmark does not reproduce in production.
• Processing scripts not versioned with the data. A threshold changed in a notebook six months ago is indistinguishable from a data change when the metrics move.

What this does not tell you

1. No universal thresholds. We deliberately quote no similarity cut-off or minimum document length. Those are workload-specific, and a number lifted from a page like this one into a pipeline is a number nobody owns.
2. Not legal advice. Whether a processing step or a cross-border transfer is permitted turns on your licence and your jurisdictions, and belongs with qualified counsel.
3. Records, not full-text guarantees. We describe a corpus of more than 160 million publicly available PRC court judgment records. Field completeness varies across decades of source formatting — which is why the answer to the second red flag is a distribution table rather than an adjective, including when the supplier is us.
4. Corpus hygiene is not model quality. Everything above removes reasons for a result to be wrong. None of it supplies a reason for a result to be right; that still comes from the task design and the evaluation you built to be hard.

For the decisions upstream of this page: license versus scrape covers build-versus-buy, the source portal's robots.txt, parsed line by line is worth reading before anyone budgets a crawler, and the academic-benchmark comparison explains why a research dataset that works for a paper does not carry a product. For the licensing mechanics, see the 2026 licensing guide. Downstream, the retrieval pipeline walkthrough covers chunking, hybrid indexing and citation grounding for the serving side, and the MCP server and English-language API cover access shape.