The Phrase Query That Returns Zero: Nine of Our Forty Largest Causes of Action, and Five of Seven Company Names

Yesterday we published a false-positive rate: 289 of 16,398 records that our index returns for one Chinese phrase do not contain it, because punctuation is discarded at analysis and token positions close up. The obvious next job was to find out whether 1.76 per cent generalises. We took ten phrases a customer would actually send us — three doctrinal terms, two causes of action, three statute-article references and two boilerplate strings — and checked every hit against the stored text.

It does not generalise. Across 2,008 sampled hits there were zero false positives. What the same run found instead is considerably worse, and it runs in the opposite direction: five of the ten phrases return exactly zero from a zero-slop match_phrase, while the strings themselves occur in millions of documents. One of them, jianshe gongcheng shigong hetong jiufen (construction project works contract disputes), is a top-ten cause of action with 827,292 records carrying it as their own cause label.

We then measured how wide the hole is. Against the forty most common cause-of-action labels in our index, nine return exactly zero on a default phrase query. Those nine causes label 10,138,333 records. Everything below was measured on 20 September 2026 against 160,291,678 records in fifteen ingest batches.

What a zero looks like from the outside

A retrieval failure that returns an error is an inconvenience. A retrieval failure that returns 0 is a false finding, because 0 is a perfectly plausible answer that a reader has no way to distinguish from the truth. Here are the nine causes, with the population that carries each label in the structured cause field, and three counts against the full text of every document in the corpus.

The first column and the third are measuring different things and should not be equal — a document can mention a cause of action without being filed under it, so the full-text count is normally the larger. On the nine rows above the full-text count is not larger. It is nothing.

For contrast, here are five of the largest causes that behave normally. Note that on two of them the default query returns more than the corrected one. Neither analyzer dominates; that matters for the fix.

Why: two tokenisers that do not agree on where the words are

The mapping on our full-text fields reads, in full:

"body":  {"type":"text","analyzer":"ik_max_word","search_analyzer":"ik_smart"}
"title": {"type":"text","analyzer":"ik_max_word","search_analyzer":"ik_smart"}

This is deliberate and, for keyword-style search, sensible. IK's max_word mode over-segments at index time so that a short query has many ways to hit; ik_smart segments the query coarsely so that it does not fragment into noise. For a match query, which only asks whether the terms are present, the asymmetry is a pure recall win.

A match_phrase does not ask whether the terms are present. It asks whether they sit at consecutive positions. And the two modes disagree about what the positions are:

construction project works contract dispute
  index (ik_max_word):  jianshe@0  gongcheng-shigong@1  gongcheng@2  shigong@3
                        hetong-jiufen@4  hetong@5  jiufen@6
  query (ik_smart):     jianshe@0  gongcheng-shigong@1  hetong-jiufen@2

the query wants hetong-jiufen at position 2; the index put it at position 4  ->  no match

The over-segmentation inserts two extra tokens between the second and third element of the query, so the phrase the query describes does not exist anywhere in the index, in any document, ever. The count is not approximately zero. It is exactly zero, in a corpus where the string occurs more than a million and a half times.

The same mechanism explains the statute-article references, where IK's number handling over-segments even harder:

An evaluation harness that scores a legal-QA model on statute retrieval, and that builds its ground truth with a phrase query against a corpus configured this way, would record an empty gold set for Article 147 and never notice.

It is not a precision problem: 2,126 of 2,126

Before recommending the corrected query we had to establish that its extra hits are real, and not the analyzer running loose in the other direction. For each of seven zero-returning strings we drew roughly 300 records from the corrected query, stratified across every ingest batch in proportion to its population, seed 20260920, and checked the stored text for the literal string.

2,126 of 2,126. The documents were always there and always contained the string. The default query was not being cautious; it was blind.

A second, independent estimate agrees. Taking the non-phrase AND query for the construction-contract term and subtracting the phrase hits leaves 2,349,208 records; of 402 sampled, 300 contain the literal string, which puts the true population near 1.75 million against the corrected query's 1,566,464. Two methods that share no machinery land within about twelve per cent of each other.

Slop is not the fix

The reflex is to add slop and move on. The last column of the first table is why that is a bad idea. Of the nine broken causes, slop=2 recovers six to a plausible figure. It leaves the house-sale contract cause at 124 hits against a true population above two million. And it leaves the motor vehicle traffic accident cause — the largest of the nine, 3.8 million labelled records — at exactly zero.

Slop also loosens every phrase that was already working, which is the opposite of what someone reaching for a phrase query wants. It converts a silent recall failure into a silent precision failure.

What does work is making the query analyzer match the index analyzer, or running both and taking the union. We measured the union on four phrases; in every case it equalled the larger of the two counts and cost nothing:

On the two working phrases the corrected query is a strict subset of the default one, so the union changes nothing; on the two broken ones the default contributes nothing and the union is the corrected count. Running both costs a second query and removes the failure mode. The cleaner long-term answer is a sub-field indexed and searched with the same analyzer, which we have not done because it is a re-index of 413 GB and we are not going to claim a fix we have not shipped.

The case this breaks worst is the one phrase search exists for

Phrase search matters most for names: a conflict check, a portfolio-exposure query, a due-diligence sweep for every judgment naming one counterparty. PRC entity names are long compounds, which is precisely the input that makes the two tokenisers diverge. We tested seven real names on 20 September 2026.

Five of seven return zero. A counterparty screen for the full registered name of China's largest commercial bank returns a clean sheet, and the honest answer is 1,113,936 documents. The two names that work are the two short enough that ik_max_word and ik_smart happen to agree.

The title field carries the same mapping and the same defect. A search for judgments whose case title contains the construction-contract cause returns 0 by default and 1,326,264 corrected; the motor vehicle cause returns 0 against 6,338,730; house-sale contracts 0 against 2,325,269. Private lending, which tokenises identically under both modes, returns 18,249,494 either way.

Even when it works, it leaks a little

One more measurement, because a phrase query that returns a large number is not thereby correct. For three phrases that do work we took the non-phrase AND set, removed the phrase hits, and sampled the remainder for the literal string:

The private-lending phrase returns 19,831,056 records and misses an estimated 69,721 more that literally contain it — a 0.35 per cent blind spot on a query nobody would suspect. The other two leak nothing measurable. As with yesterday's false-positive rate, this is a per-phrase property and cannot be extrapolated from one term to the next. That is the general lesson of both days: in Chinese full-text retrieval the error rate is a property of the string, not of the system, so a single benchmark number for a corpus is close to meaningless.

What we are doing about it

Three things, stated as what they are rather than as a roadmap.

1. The mapping is unchanged as of this writing. Changing a search analyzer alters what every existing client's query returns, including queries whose current answers customers have already relied on, and we will not do that silently on a Sunday. The defect is documented here first.
2. We are auditing which of our own endpoints route to a phrase query. Our public keyword search endpoint does not: a plain q= for the construction-contract cause returns a normal capped result set, so keyword search is unaffected. The exposure is on paths that promote a query to an exact phrase, which is exactly what a full institution name does on our MCP endpoint.
3. Any count we have previously given a customer as an exact-phrase count is being re-run with the union query. If a number moves, the customer gets the new number and the reason, unprompted.

What a buyer should run before signing

This takes about ten minutes against any Chinese case-law platform that exposes a phrase or exact-match mode, and it does not require access to the mapping.

1. Take the platform's own ten largest causes of action from whatever facet or filter list it publishes.
2. Run each one as an exact phrase against the full text. Any that return exactly zero are suspects — a real corpus does not contain zero documents mentioning its own most common cause of action.
3. Re-run each suspect as an ordinary keyword search. If that returns millions, the zero was an artefact and the platform's phrase mode cannot be trusted for recall.
4. Repeat with three long entity names: a bank's full registered name, a district court's full name, and a technology company's full name. This is the test that failed five times out of seven for us.
5. Ask the vendor, in writing, for the index analyzer and the search analyzer on the full-text field. If they differ, ask for the zero-return rate over the platform's top fifty causes of action.

We publish our own failures on this blog because the alternative — a buyer discovering them after signature — is worse for us than the embarrassment. Every defect in this post is reproducible against a trial slice using the queries above.

What this does not show

1. It does not show that our corpus is missing documents. Every measurement here says the opposite: the records exist, contain the literal string, and are returned the moment the query is expressed correctly.
2. It does not establish a corpus-wide error rate. We measured 40 cause labels, 7 entity names, 3 statute forms and 10 hand-picked phrases. Nine of forty is a rate for that list of forty, not for the language.
3. The 10,138,333 figure counts records labelled with the nine broken causes. It is a measure of how much of the corpus a user is likely to be asking about, not of how many documents each broken query fails to return.
4. The residual estimates in the leak table come from stratified samples of roughly 400 with a minimum of two records per ingest batch, which slightly over-weights small batches; the weighted and unweighted rates agreed to within 0.01 percentage points, so we report the weighted one.
5. We have not tested other analyzers (jieba, HanLP, ICU) or other engines. The mechanism — positional disagreement between an index-time and a search-time tokeniser — is not specific to IK, but the rates here are.
6. Nothing here revises any outcome statistic we have published. No grant rate, claim-type rate or coverage figure in our earlier posts was produced with a phrase query of this shape.