Traditional and Simplified Chinese in Case Law Retrieval: Why “We Support Both” Is Two Questions

Someone in Hong Kong, Taipei or Singapore types a search term in traditional characters into a demo of a mainland judgment corpus. Results come back. Nobody in the room reads enough of them to know whether they are the right results, the demo moves on, and “supports traditional characters” goes into the evaluation notes as a yes.

That single sentence is carrying two claims that fail independently. Can the system find the documents, and did it find the right ones? The first failure is loud: an empty result set is impossible to miss. The second is the one that survives procurement, because a script mismatch in Chinese retrieval does not raise an error. It returns a smaller pile that looks exactly like an answer.

The coverage verification protocol argues that coverage claims have to be tested rather than read. This page is the same argument applied to one line item that almost never gets tested, because everyone assumes it was solved somewhere upstream by a standard.

Unicode does not do this for you

The first assumption to dispose of is that text normalisation is a solved, standard-supplied step. It is, for some problems. Unicode defines four normalisation forms, and the compatibility ones genuinely help with Chinese legal text: NFKC folds full-width Latin letters, full-width digits and full-width parentheses down to their ASCII equivalents, which matters more than it sounds like it should, because PRC docket numbers are routinely written with full-width brackets in one system and half-width in another.

What no normalisation form does is fold script. Run the traditional forms of characters like fa (發), tai (臺) or yue (粵) through NFC, NFD, NFKC and NFKD and all four return them unchanged. Traditional and simplified characters are separate code points with no compatibility relationship, and that is a deliberate design decision in the standard, not an oversight.

So a docket number written in traditional script and full-width brackets needs two different repairs, from two different layers. NFKC fixes the brackets and leaves 粵 alone. The script fold has to come from somewhere else: a mapping table, maintained by someone, carrying a version number, with a defensible position on the cases below. Whenever a supplier answers the normalisation question with the word “Unicode”, you have learned that the question was heard as a question about encoding.

The fold is many-to-one going one way and a guess going the other

The second assumption to dispose of is that the two scripts are two spellings of the same thing. In the direction most pipelines run — traditional to simplified — the mapping is many-to-one. Taking OpenCC's traditional-to-simplified table over the main CJK block, 140 simplified characters have at least two distinct traditional sources, and 13 have three or more. The simplified 干 stands in for 乾, 幹 and 榦; 复 covers both 復 and 複; 台 absorbs 臺, 檯 and 颱.

Folding to simplified is therefore deterministic and lossy. Expanding to traditional is neither: it has to pick a sense, and it can pick wrong. That gives the single most useful demonstration on this page — the round trip is not the identity. Take the traditional 行政復議, administrative reconsideration. Fold it to simplified and you get 行政复议, correctly. Expand that back to traditional and you get 行政複議, where the second character now means duplication. The word came back as a term that does not exist in administrative law.

The consequence is structural, not cosmetic. Any design that folds text on ingest and unfolds it for display is capable of returning a judgment that reads plausibly and contains characters the court never wrote. For a legal user, that is a worse failure than returning nothing, because the artefact of the pipeline is indistinguishable from the document.

The characters at stake are the ones you look things up by

If script variation only affected prose, an imperfect fold would cost some recall in the body text and little else. It does not. It lands squarely on the two fields with the highest lookup value in Chinese legal data.

Docket numbers

A PRC case number is assembled from a year, a court code built on a provincial abbreviation, a case-type marker and a sequence number. Of the 31 provincial-level abbreviations, 11 are written differently in traditional script — 粤/粵, 沪/滬, 苏/蘇, 鲁/魯, 辽/遼, 陕/陝, 闽/閩, 赣/贛, 琼/瓊, 宁/寧 and 晋/晉 — and the trailing markers differ too: 终/終, 号/號. One docket number can therefore differ from its traditional rendering in three or four positions, plus bracket width:

(2021)粤01民终12345号     <- as published
（2021）粵01民終12345號   <- as a Hong Kong associate might type it

Exact-match lookup on a case number is the one query where users expect zero ambiguity, and it is precisely where the failure is invisible: the system returns no result, the user assumes the case is not in the corpus, and nothing anywhere logs that a supported query form was not supported.

Anonymised party names

This one is worse, and it is the reason to be careful about which direction you convert. Published judgments anonymise natural persons to a surname plus a placeholder character — 范某, 余某. Both of those surnames are written identically in traditional script. But each has an unrelated look-alike that is a traditional form of something else: 範 (a pattern or model) and 餘 (surplus). Every OpenCC simplified-to-traditional configuration tested for this page, including the region-specific and phrase-aware ones, converts 范某 to 範某 and 余某 to 餘某.

So a well-meaning “traditional-friendly” layer that converts the corpus, or the user's query, in that direction will quietly rewrite real party names into names that do not exist. The affected surnames are common, and a party name is often the only handle a user has on a case they already know about.

Four places to put the fold, and what each one costs

There is no free placement. The useful question in a technical call is not whether a supplier normalises, but which of these costs they have chosen and whether they know they chose it.

A supplier who can name their placement and its cost in one sentence has thought about it. The answer that should stop the call is the one that describes the search box rather than the pipeline — “yes, you can type either” — because that is a statement about what the input field accepts, not about what happens to the text afterwards.

The asymmetry that survives every demo: one product, two code paths

Here is the failure mode worth budgeting an hour to test, because it is invisible in a scripted demo and it does not require anyone to have been careless.

Structured filters and free-text search usually travel different roads. A filter on province, court or cause of action resolves against a controlled vocabulary, and that vocabulary gets normalised once, at ingest, by whoever built the taxonomy. Free-text search goes through an analyser chain, and script folding is an optional component in that chain that has to be switched on deliberately. Nothing forces the two roads to agree. A single product can therefore be completely script-agnostic in its filters and completely script-literal in its search box.

The measurable signature is specific enough to test for in minutes: the filter returns identical totals for both scripts, while the same concept typed into free-text returns a much smaller and qualitatively different set for the traditional form — different enough that the leading result may not even be the right kind of case. This is not hypothetical. It is a pattern we have measured in a system we operate, and it is not currently fixed; the disclosure at the end of this page says what that does and does not mean.

Why does it survive? Because both paths return something. The filter demo looks flawless. The search box returns a list, and an evaluator who cannot fluently read results in the second script has no way to see that it is the wrong list. It fails by returning results, not by returning an error — and every quality gate in a normal procurement is built to catch errors.

The ten-minute test

Run this against any supplier, in a shared screen, before the pilot rather than during it. It needs no credentials beyond a trial key and no knowledge of the vendor's stack.

1. Same concept, both scripts, free text. Pick a term whose two forms differ in more than one character — 劳动争议 and 勞動爭議 will do. Compare totals and the identity of the top three results. Equal totals with different leading cases is still a failure; so is a total that is quietly capped.
2. Same concept through the structured filter. Then ask the two paths to agree with each other. If they disagree, ask which one is authoritative, and what a user is supposed to conclude when the filter and the search box report different worlds.
3. One docket number, four ways. Simplified and traditional, full-width and half-width brackets. Any combination that misses tells you which of the two repairs is missing.
4. One anonymised party name in traditional script. Then check what the system did with the surname. If it echoes your query back with 範 or 餘 where you typed 范 or 余, the conversion is running in the dangerous direction.
5. Ask what is stored, not what is searched. Request the full text of one record as delivered, and compare it against the same document at the public source. This is the only check that catches an ingest-time fold, and it is the one that matters for citation.
6. Ask which mapping table, which version, and what happens on upgrade. A supplier who answers with a library name and a version has an answer. A supplier who does not, does not have a mapping table — they have a default.

Two habits make this test worth more than the ten minutes. Run it against a corpus slice you can read, so “wrong results” is a judgement you make rather than one you accept. And record the answers in writing, because this is exactly the class of behaviour that changes silently between releases — the kind of thing a change log should be reporting to you and usually is not.

Reproduce the numbers

Everything quantitative on this page comes from the two lines below. It is worth running: the specific characters that matter to your practice area are not the ones in this article.

import unicodedata, opencc
t2s = opencc.OpenCC('t2s'); s2t = opencc.OpenCC('s2t')

# 1. no Unicode form folds script
all(unicodedata.normalize(f, '發') == '發' for f in ('NFC','NFD','NFKC','NFKD'))   # True
unicodedata.normalize('NFKC', '（2021）')                                          # '(2021)'  brackets fixed, script untouched

# 2. how many-to-one the fold is
src = {}
for cp in range(0x4E00, 0x9FFF):
    ch = chr(cp); s = t2s.convert(ch)
    if s != ch and len(s) == 1: src.setdefault(s, set()).add(ch)
len([k for k, v in src.items() if len(v) > 1])        # 140   (13 of them have three or more)

# 3. the round trip is not the identity
s2t.convert(t2s.convert('行政復議'))  # reconsideration -> duplication
s2t.convert('范某'), s2t.convert('余某')   # two surnames rewritten into different characters

What this page does not settle

It is not a claim that we pass the test. The structured-versus-free-text asymmetry described above is one we measured inside a system we operate, and it is open. Nothing here should be read as a capability statement; the test is published because it is the right test, not because we are finished with it.

The numbers are one table's opinion. OpenCC is a widely used mapping, not an authority, and its own regional configurations disagree with one another about individual characters. A different table, or a different version of the same table, will give different counts. That is the point of asking a supplier which one they use.

No claim is made about how often this occurs elsewhere. We have not tested another supplier's retrieval layer and have no sample from which to say that this is common, rare, or anything in between. The argument here is that the defect is cheap to test for and expensive to discover late — not that anyone in particular has it.

Nothing here is about licence scope. Whether traditional-script material sits inside or outside a given agreement is a drafting question, not a retrieval one, and belongs with the pilot scope rather than with the analyser chain.