How Many Tokens Are in 160 Million Chinese Court Judgments? A Measured Count

The first question a model team asks about a Chinese case law corpus is not how many judgments it holds. It is how many tokens. Compute budgets, data-mix ratios and, increasingly, licence fees are denominated in tokens, and a record count does not convert into one without a measurement nobody publishes.

So we measured our own. On 27 September 2026 we drew a stratified sample from every one of the fifteen ingest batches in our index of 160,291,678 PRC judgment records, counted the body text with two production tokenizers, removed the text that is identical in every judgment, and checked each sampled record against every other record carrying the same case number. The answer is not one number. It is a funnel, and the tokenizer you pick moves the top of it by 43 per cent before any cleaning happens.

The rest of this page shows how each line was measured, where the error bars come from, which parts of the answer depend on choices a buyer should make rather than accept, and what the count still does not tell you. Everything here describes our own index on the stated date. It is not a claim about any other vendor's corpus, and not a census of Chinese court judgments — nobody can state that universe's size, so nobody can state a share of it.

How the count was made

A token count over 160 million records is an estimate unless you tokenize all of them, and tokenizing all of them to answer a sizing question is the wrong trade. The estimate is only as good as the sample, and we have learned the hard way that a sample from a large search index is less random than it looks.

• Stratified by ingest batch. Our index merges fifteen batches with very different histories: one holds 40.7 million records, one holds 7,000. A seeded random_score query over the whole index is not a random sample, because the seed is applied per shard, and on 18 September a 1,500-record “random” sample from us came entirely from two of the fifteen batches. So we drew 500 records from each batch separately (random_score, seed 20260927, field _seq_no) and weighted each batch's mean by its record count.
• Two tokenizers, same text. Every sampled body was encoded with tiktoken 0.13.0 using o200k_base and cl100k_base. No normalisation was applied before counting the raw line: newlines, full-width spaces and the spaced-out document titles courts like to print (民 事 判 决 书) were all counted, because they are in the text a buyer receives.
• Error bars from the design, not a guess. The confidence intervals above are the standard stratified-sampling interval: the per-batch variance of tokens per record, scaled by batch size and summed. Record lengths are heavily skewed, so the interval is wider than a record count's would be. That is a property of the data, and a sizing figure quoted without one should be treated as a point estimate of unknown quality.
• Body text only. Titles, case numbers, court names, dates and other metadata fields are excluded. They add a small amount of text per record, and whether they belong in a training mix is a separate decision.

The sample totals are internally consistent: 7,500 records, an estimated 210.4 billion characters of body text (95% CI 199.9–220.9 B), and a mean of 1,543 characters per record that contains a judgment.

The tokenizer moves the number by 43 per cent

The same 210 billion characters are 165 billion tokens under o200k_base and 237 billion under cl100k_base. Put per character: the newer encoding averages 1.28 characters per token on this corpus; the older one averages 0.89, which means it emits more tokens than there are characters. That is not a quirk of legal text. Byte-level encodings that learned their merges mostly from English split a large share of Chinese characters into two or more byte-level pieces, and a Chinese court judgment is dense with characters those vocabularies saw rarely: court names, legal terms of art, party names, the full-width punctuation of formal Chinese.

Three consequences follow for anyone sizing or pricing this data.

1. A token count without a named tokenizer is not a number. The same corpus is “165 billion tokens” or “237 billion tokens” depending on which vocabulary did the counting. A vendor quoting the larger figure and a buyer budgeting on the smaller one are looking at identical data.
2. Your own tokenizer is the one that matters. Model families built with large Chinese vocabularies will likely land below both figures. We have not measured them, so we do not quote them; the method below takes an afternoon to rerun with any tokenizer you can load.
3. Per-token pricing silently re-prices when the tokenizer changes. If a licence fee is written per token and the tokenizer is not fixed in the contract, a later switch of vocabulary changes the invoice without changing a byte of data.

Where the tokens actually sit

A record count invites you to picture 160 million documents of similar size. They are not. Among records that contain a judgment, the median body is 824 characters; the 90th percentile is 3,673; the 99th is 9,402. The longest tenth of those documents hold 39 per cent of all tokens. Short enforcement rulings and procedural decisions are numerous but light; the long first-instance and appellate judgments that carry most of the reasoning are fewer and heavy.

The batches differ just as much, and that is where a quote built on an average goes wrong:

n = 500 records per batch for the text and token columns; n = 120 text-bearing records per batch for the duplicate column. Every figure is an estimate with its own sampling error; per-batch figures are less precise than the corpus totals.

Two batches carry 54 per cent of all tokens. One batch of 4.4 million records carries none: its body field is empty almost throughout, a defect we documented earlier and which a field-presence check does not reveal (“Has Full Text” Is Three Numbers). Another holds 11.5 million records whose body is a one-line statutory reason for non-publication. Those placeholder records are 7.17 per cent of the corpus by count and, by our sample, about 0.06 per cent of it by tokens — roughly 0.1 billion. They are a record-count problem, not a token-budget problem, which is why a token count and a record count of the same delivery can tell such different stories about quality.

A note on consistency with that earlier page: today's sample puts placeholder records at 5.94 per cent of the corpus, below the exact count of 7.17 per cent. The exact count is right; the gap is sampling error, concentrated in the one batch where nearly all placeholders sit. It is a useful reminder that a stratified estimate of a rare category is noisier than it looks, and that anything you can count exactly, you should.

About a tenth of the text is the same in every judgment

PRC judgments are written to a strict template, and parts of that template are reproduced almost word for word in millions of documents. For a retrieval index this is mostly harmless. For pretraining it is not: a model sees the same sentences millions of times and learns to produce them fluently, and the tokens spent on them are tokens not spent on reasoning. We measured six kinds of formulaic span with fixed patterns and counted the characters they cover in every text-bearing record:

Three cautions before you strip any of this.

• It is a lower bound. Our patterns catch the common phrasings. Courts vary them, and older documents vary them more. The real formulaic share is higher, and in categories dominated by mass filings it is far higher, because entire documents are near-copies of one another (mass-filing clusters). Those are not measured here.
• Formulaic is not the same as empty. A fee sentence has a fixed shape but carries a ruling: who pays, and how much. The late-payment notice is the reason a common keyword looks topical when it is not — “加倍支付” appears in about 21 per cent of all records in our index, mostly through this one sentence. Strip it for pretraining, keep it for extraction.
• Strip by section, not by a regex over the whole document. The same wording occasionally appears inside the court's reasoning, quoted or paraphrased. The safe approach anchors each pattern to the section of the judgment where it belongs, as we describe in preparing Chinese court judgments as a training corpus.

Six per cent of the tokens are second copies

A multi-batch corpus assembled over years will hold the same judgment more than once. The question for sizing is not how many records share a case number — many legitimately do, because one case produces several documents — but how much of the text is a verbatim repeat.

We took 120 text-bearing records from each batch, looked up every other record in the index with the same case number, and compared bodies after removing whitespace. For each sampled record with k identical copies in the index, a fraction (k−1)/k of its tokens is redundant: keep one copy, and the rest are surplus. Averaging that fraction over the sample, token-weighted and batch-weighted, gives an unbiased estimate of the redundant share without assembling the whole duplicate graph.

• 25.2% of records share their case number with at least one other record.
• 13.2% of records have at least one verbatim copy elsewhere in the index.
• 6.3% of tokens (bootstrap 95% CI 5.3–7.5%) are redundant copies. Ignoring punctuation as well as whitespace barely changes it (6.4%).
• Of the sampled records that had a copy, 82% (512 of 622) had it in a different ingest batch. Duplication here is overwhelmingly a by-product of overlapping collection windows, not of a single batch repeating itself — which is why the rate varies from zero in some batches to over half in others (table above).

The gap between 25.2 and 13.2 per cent matters. Deduplicating on case number alone would delete genuine distinct documents — a judgment and the enforcement ruling that follows it, a ruling and its correction — and overstate redundancy by roughly a factor of two. We cover the survivorship question, which copy to keep and why, in deduplication survivorship rules.

What a per-token licence should pin down

If a licence fee, a delivery milestone or an acceptance criterion is expressed in tokens, the measurements above say what the contract has to fix. None of this is legal advice; it is the list of parameters that changed our own answer.

1. The tokenizer, by name and version. On this corpus the choice alone is a 43 per cent swing.
2. The text unit. Body only, or body plus title and metadata; raw, or after an agreed normalisation of whitespace.
3. The boilerplate policy. Whether formulaic spans count, and under which written pattern list. A 10 per cent difference is large enough to argue about later.
4. The duplicate rule. Verbatim text within the same case number, not case number alone, and which copy survives.
5. Exclusion of non-documents. Empty bodies and non-publication placeholders should count as zero; by tokens they are close to zero anyway, but they should not be billable records either.
6. The measurement method. Stratified by batch, with a fixed seed and stated sample sizes, and an interval rather than a point figure. A single number with no interval invites a dispute you cannot settle.

How those parameters translate into commercial terms is covered in what Chinese court data licensing costs; how to verify them on a delivery is in ten acceptance checks.

Reproduce it on any delivery

The whole measurement fits in a short script. Sample each batch separately, encode, weight, and report an interval.

import tiktoken, statistics as st
enc = tiktoken.get_encoding("o200k_base")   # or your own tokenizer

# sample[batch] = list of body strings, n per batch, fixed seed
# count[batch]  = number of records in the batch
total, var = 0.0, 0.0
for b, bodies in sample.items():
    t = [len(enc.encode(x or "")) for x in bodies]
    total += count[b] * st.mean(t)
    var   += count[b]**2 * st.pvariance(t) / len(t)
print(f"{total/1e9:.1f}B tokens, 95% CI ±{1.96*var**.5/1e9:.1f}B")

For the duplicate share, look up each sampled record's case number, count the verbatim copies k after stripping whitespace, and average tokens * (k-1)/k in the same weighted way. In Elasticsearch, draw each batch's sample with a function_score query filtered to that batch, using random_score with an explicit seed and field: "_seq_no"; never sample the whole index in one query.

What this count does not tell you

• Near-duplicates across different case numbers. Mass filings by one plaintiff against hundreds of defendants produce documents that differ only in names and amounts. They are not counted as duplicates here, and in some categories they dominate.
• Counts under Chinese-heavy tokenizers. We measured two widely used encodings. Tokenizers with large Chinese vocabularies will likely give lower counts; we have not measured them and do not quote a figure.
• Quality. A token is not a useful token. Truncation, misattributed document types and batch-specific artefacts affect what those tokens teach a model; see the fields nobody audits.
• Anything outside our index. Every number here is a measurement of our own corpus on 27 September 2026. It is not a statement about another vendor's data, and not an estimate of all judgments ever published by Chinese courts.