Can You Crawl Chinese Court Judgments for LLM Training? What robots.txt Actually Says

Somewhere in the planning of every China-coverage feature, an engineer says the reasonable thing: the judgments are published by the courts, so let's just crawl them. It is a fair proposal and it deserves a factual answer rather than a vendor's shrug. The factual answer happens to live in a single 982-byte file that anyone can fetch in ten seconds, and almost nobody on these calls has read it.

So this piece does something narrow and, we hope, useful: it reads that file line by line, checks it against the crawler documentation that OpenAI, Anthropic, Google and Perplexity publish about their own bots, and then—this part matters more than the rest—states plainly the four things none of it proves. Every factual claim below is a URL you can retrieve yourself. Where we are uncertain, we say so.

The file, parsed line by line

Retrieved 10 August 2026 from https://wenshu.court.gov.cn/robots.txt: HTTP 200, 982 bytes, SHA-256 beginning 0c97ffd2f7ae. It contains 20 User-agent groups, and they fall into exactly three buckets.

Reproduce it in one line, and compare the hash against ours:

curl -s https://wenshu.court.gov.cn/robots.txt | shasum -a 256
# 0c97ffd2f7ae2ec7f72b3b9401fd75162b59f1bf536b56515585741acbc970f8

A short note on method, because the number is easy to get wrong and we got it wrong ourselves in an internal draft: 19 is the count of named user agents (15 + 4), not the count of allowances. If you see "19 crawlers allowed" anywhere, including in an earlier version of our own notes, it is a miscount.

Which AI crawlers are named? None of them.

We grepped the file, case-insensitively, for every AI-associated user-agent token we could think of. The result is uniform:

Fourteen tokens, zero appearances. Under the standard longest-match convention, each therefore falls into the final User-agent: * group, whose sole directive is Disallow: /. That is the whole finding. It is small, it is checkable, and it is the single most load-bearing fact in the build-versus-buy conversation about Chinese case law—which is why it is strange that it circulates so rarely.

The distinction almost everyone collapses: three kinds of bot

Here is where discussions of this topic usually go wrong, including ours before we sat down with the primary sources. "AI crawler" is not one thing. The fourteen tokens above belong to three functionally different classes, and the vendors themselves are precise about the difference in their own documentation.

Two consequences follow, and they cut in opposite directions.

First, the honest version of the finding is broader than "no training crawler is allowed." None of the three classes is named, so on the documented policies, neither the training route nor the answer-grounding route reaches this site automatically.

Second—and this is the correction we most want to make to the sloppy version of this argument—you cannot say Perplexity has promised not to train on this data. Its bot was never a training crawler in the first place; the company says so on its own documentation page. Any buyer's counsel will open that page and find the sentence inside a minute, and the rest of your analysis dies with it. Say the precise thing instead.

As for the vendors' general posture: Anthropic states that "Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt." Google states that its common crawlers "always obey robots.txt rules when crawling automatically," and separately that "Google-Extended doesn't have a separate HTTP request user agent string… the robots.txt user-agent token is used in a control capacity."

Four things this does not prove

An argument that only points one way is a sales pitch. These are the limits, and we would rather write them here than have a reader discover them and reasonably conclude we were hiding something.

1. robots.txt is not law. It is a voluntary convention, and the enforceability of text-and-data-mining opt-outs varies by jurisdiction. The most the sections above support is a statement about policy compliance—that a crawler following its operator's published policy will not automatically collect this site. That is not the same as a statement about legality, and we are not making one.
2. This site is open to search indexing. Googlebot and MSNBot are explicitly allowed, along with the major Chinese engines. Anyone who writes "the Chinese court portal blocks crawlers" is wrong and will be corrected in one reply. The accurate framing is narrower and more interesting: search indexing is open; automated collection for AI training or grounding is not invited.
3. User-triggered retrieval is a documented exception. An assistant can fetch a page for a user who asks for it, and the vendors say robots.txt may not govern that. The difference that matters is between one page and a corpus—deduplicated, field-normalised, and queryable across a hundred million-plus records. No user-triggered fetcher assembles the second thing.
4. No accusation, and no claim of exclusivity. We have no evidence that any company has crawled this site, we name none, and we do not claim our own corpus is the only lawful route to PRC case law. Those would be different assertions requiring different evidence, and we do not have it.

What this means for an engineering plan

Strip out the compliance layer and a practical problem remains, which we have costed out in detail elsewhere: the crawler is roughly a tenth of the work. The other nine tenths are parsing formats that differ by court and by decade, deduplicating documents that appear in several places at once, and deriving stable fields—case number, court, date, cause of action, parties, outcome—on every record, so that filtering, retrieval and citation are possible at all. That pipeline does not finish; it is a standing tax on your roadmap, and it competes with the product your customers are actually paying for.

Coverage compounds the problem. Public availability of PRC judgments peaked in 2020 and fell sharply through 2023, after which the comprehensive database moved to the courts' internal network with a smaller curated library remaining public. A crawler collects what is reachable today, which is a slice weighted by year and category—and volume disguises this beautifully. Millions of documents feel comprehensive right up until a user asks about a 2022 dispute and your index has nothing on point.

The free alternatives, described fairly

The honest comparison is not against a strawman. The Chinese AI and Law challenge datasets are real, they are good, and for a defined set of purposes you should simply use them instead of paying anyone.

CAIL2018 contains, in the authors' words, "more than 2.6 million criminal cases published by the Supreme People's Court of China" (arXiv:1807.02478), and the GitHub repository carries an MIT licence. If your work is Chinese criminal judgment prediction, charge classification, or benchmarking, that is a strong and free starting point, and no licensing conversation is required.

The limits are scope and licensing rather than quality. Checked via the GitHub API on 10 August 2026:

Two things to take from that table. The obvious one: four of the five carry no licence file, and "no licence" is not a permissive default—it is an absence, which your counsel will treat as an open question rather than a green light. The less obvious one applies far beyond these repositories: a repository licence is not a dataset licence. An MIT header can cover the loader scripts while the documents themselves are governed by something else. Anyone assembling training data from public repositories should check which artifact a licence file actually attaches to.

And the scope limit is simply structural: these corpora are criminal-only, Chinese-only, and frozen at their release year. If your product needs contract, labour, IP, administrative or company disputes; needs English-language fields; or needs anything after the freeze date, the academic sets do not reach it—not because they are deficient, but because they were built to answer a research question, not to ship inside a commercial product.

What is actually left to buy

Put the crawler question and the academic-dataset question side by side and the remaining gap is narrow and specific. It is three things:

• Scale and cross-category coverage. Assembling a corpus that spans causes of action is a different problem from fetching documents, and it is the one that does not yield to a sprint.
• Structure. Stable, normalised fields on every record, so retrieval, filtering and verifiable citation are possible without an NLP project of your own.
• Licensing certainty. Written terms covering AI training, retrieval and indexing—the sentence your own enterprise customers will ask you to produce in a due diligence questionnaire, where "we license it under written terms" and "we scraped it" land very differently.

That is the category we occupy. SinoVerdict licenses a structured, machine-readable corpus of more than 160 million publicly available PRC court judgment records, sourced from publicly available PRC court rulings (China Judgments Online and provincial portals), normalised into structured fields as a snapshot through 2023 with incremental updates available on request, and delivered as a bulk dataset, a REST API and an MCP server. For deal mechanics and a buyer's checklist, see our guide to licensing Chinese court judgment data.

Sources

Every claim above, with its source and retrieval date. All four vendor pages returned HTTP 200 with no login required on 10 August 2026, and the quoted sentences were checked verbatim against the retrieved text.