What Does a Cross-Border Retrieval Path Cost You? Latency Budgets and Per-Query Economics for a China-Hosted Case Law Corpus

The question arrives in almost every technical evaluation, and it is framed wrongly: what is the latency of your API?

Latency is not a property of an API. It is a property of a path, a payload, a connection strategy and — the term teams most often omit — a count of how many times a single user question crosses the border before an answer comes back. Two engineering teams can integrate the identical endpoint and land an order of magnitude apart, because one of them makes a single pooled request that returns identifiers and snippets, and the other opens a fresh connection four times in sequence and pulls full judgment text on every hop.

Two earlier pieces on this site end at the door of this problem. The one on deployment geography establishes that a query-only topology costs you per-query latency on every request; the one on pilot design tells you to run the evaluation on the topology you will actually operate. Both say measure it. Neither says how. This page is the engineering answer: what the floor is, where the milliseconds go, what a query costs in the units your finance team uses, and which interventions are worth the work.

The floor is physics, and you can compute it before any call

Before you ask a supplier anything, you can establish a number they cannot beat. Light in single-mode fibre travels at roughly two-thirds of its vacuum speed, which reduces to a rule of thumb worth memorising: about one millisecond of round-trip time for every hundred kilometres of fibre. Apply it to great-circle distance from a corpus hosted in Shanghai:

These are floors, not estimates. Fibre follows cable routes rather than great-circle paths, and every switch, peering point and queue adds delay on top. Measured round-trip time on these paths is always higher than the table, sometimes considerably, and by a ratio that is specific to your route and your providers. That ratio is a thing to measure rather than assume.

What the floor is good for is a sanity check that costs nothing. If a product requirement says a legal research answer must return in less time than the floor for your chosen pair of regions, the requirement is not achievable by any supplier on any contract, and the conversation you need is about topology, not procurement. Better to learn that in week one than in month four.

Where a request's milliseconds actually go

Round-trip time is the unit, not the total. A single retrieval request spends time in five places, and only one of them is the vendor's to control.

1. Connection establishment. A cold request pays for the transport handshake and then the TLS handshake before a single query byte moves. On a long path those handshakes are not a rounding error; they are frequently the largest line in the whole budget, and an unpooled client pays them again on every request. It is also the cheapest defect on this page to remove.
2. Query execution on the corpus side. The part everybody asks about, and usually the smallest term once the index is warm and the query is well-formed. Full-text search over a large judgment corpus with a selective filter behaves differently from an unfiltered semantic scan, so ask which one your access pattern triggers.
3. Response transfer. A large response does not arrive in one round trip. Transfers begin conservatively and ramp up, so a payload of any size costs several round trips of wall-clock rather than one. This is why the difference between returning snippets and returning full judgment text is not a bandwidth line item — it is a latency line item.
4. Path variance. Long international routes are not merely slower than short ones; they are more variable. The median can look acceptable while the upper percentiles are what your users actually complain about. Budget on the tail.
5. Your own work. Embedding the query, reranking the hits, and generating from them. This is your compute and your latency, and it belongs in the same budget even though no vendor appears on the invoice for it.

The multiplier nobody budgets for: sequential depth

Here is the term that turns an acceptable path into an unusable product. A retrieval-augmented answer is rarely one request. A representative pipeline rewrites the user's question, retrieves candidates, fetches the full text of the top hits to ground the answer, and sometimes runs a second retrieval based on what the first one surfaced.

If each of those steps waits on the one before it, the path cost is not paid once. It is paid once per sequential step:

LATENCY PER ANSWER  (the shape, not the numbers)

    T  ~=  n_seq * RTT_measured        # border crossings in series
         +  n_seq * t_query            # corpus-side execution
         +  t_transfer(bytes, RTT)     # ramps over several RTTs
         +  t_local                    # your embed / rerank / generate

  n_seq  = SEQUENTIAL round trips per answer, not total requests.
           Requests issued in parallel cost you one n_seq, not many.

  The lever is n_seq. Halving it does more for the user than any
  plausible improvement in RTT, and it is entirely on your side of
  the boundary.

The distinction between total requests and sequential depth is the whole point. Twenty requests fired in parallel cost roughly one crossing of wall-clock time; four requests that each wait for the previous one cost four. Teams who profile their pipeline for the first time routinely find one avoidable serial hop — a metadata lookup that could have been folded into the retrieval call, or a full-text fetch issued per document instead of as a batch — and removing it is worth more than any negotiation about the endpoint.

Count your sequential depth before you benchmark anything. It is knowable from a diagram, and it multiplies every other number on this page.

What a query costs, in the units your finance team uses

Cost has the same structure as latency: the vendor's price list is one term among several, and the unit everyone quotes is the wrong one. Requests are not what your business sells. Answers are. Model cost per user-visible answer, because the ratio between the two is whatever your fan-out happens to be.

COST PER USER-VISIBLE ANSWER  (fill in your own measured inputs)

  n_req   requests per answer          (>= n_seq; includes parallel)
  b_out   bytes leaving the corpus side, per answer
  c_req   per-request charge under your licence
  c_gb    egress charge per gigabyte, on whichever side pays it
  c_comp  your own compute per answer (embed, rerank, generate)
  h       share of answers served from your local cache

  cost  ~=  (1 - h) * ( n_req * c_req  +  b_out * c_gb )  +  c_comp

  Record the DATE and the LOAD LEVEL beside every measured input.
  A figure taken at 03:00 with a single client is not the figure
  you will operate at, and it will be quoted back to you later.

Two observations matter more than the expression itself.

First, the byte term is usually the one that surprises people. Full judgment text is large, and a pipeline that pulls complete documents for every candidate rather than for the ones actually shown to a user moves a great deal of data for no user-visible benefit. Where the payload boundary sits — identifiers and snippets across the border, full text on demand — is a bigger commercial lever for most designs than the per-request price, and it is negotiable in a way physics is not.

Second, the cache share term is the one that changes the business case, because legal retrieval has a heavily concentrated head. The same landmark authorities are requested over and over across users and sessions. But local retention of judgment text is not merely an engineering choice: it is a copy of licensed material sitting on your infrastructure, which engages both the terms discussed in the licensing cost and terms piece and the placement questions in the deployment geography piece. Establish whether you are permitted to cache, and for how long, before you build a cost model that depends on it.

Six interventions, in order of effect per unit of work

Given a path you cannot shorten, this is the order we would work in. The first two are usually free, and most teams have not done them when they start negotiating about the third.

Note what is absent from this list: asking the supplier to make the endpoint faster. Corpus-side execution is real and worth asking about, but on a long path it is rarely the dominant term, and a team that has not yet pooled its connections is not in a position to know whether it is.

Note also that items 5 and 6 stop being engineering decisions partway through: both change where data rests. Exhaust items 1 through 4 first, then decide whether the remaining gap is worth a conversation with counsel.

Measuring it so the number survives production

Most latency figures presented in evaluation write-ups do not survive contact with production, for the same handful of reasons every time. They were taken from a developer laptop rather than from the serving region. They were taken at a quiet hour. They were averaged. They used a single result where the product asks for ten. They discarded the failures, which is the equivalent of reporting a flight's punctuality after excluding the cancellations.

LATENCY MEASUREMENT PLAN  (minimum viable version)

WHERE  From the region your inference will run in. Not a laptop,
       not a CI runner in some third region.
WHEN   A full daily cycle, so that both business days on the two
       sides of the path are covered.
WHAT   p50 / p95 / p99, reported SEPARATELY for:
         - warm pooled connection   vs  cold connection
         - snippet-sized response   vs  full-text response
         - your real top-k          vs  a single result
LOAD   Record concurrency beside every sample. A latency figure
       without a load figure is not a measurement.
FAIL   Count timeouts and errors as observations, not as samples
       to be discarded. On a long path the tail is the product.
OUT    One table of measured inputs feeding the cost model above,
       each row stamped with date, region, load and payload size.

Run this during the pilot, not after it. As the pilot design piece argues, the threshold has to be written down before the results exist, or the results will be interpreted by the people who want the project to proceed. A latency budget is exactly the kind of criterion that quietly relaxes once a number is on the table, and the way to prevent that is to name the number, and the person permitted to enforce it, in advance.

One further discipline: measure the same path more than once, weeks apart. International routes change — providers re-peer, capacity shifts, cable events happen — and a corridor that behaved one way in the first week of a pilot may behave differently in the fifth. This is the same calendar-time argument that governs freshness testing.

What this page does not settle

The model above is a shape, not an answer. It will not tell you what your route measures, because that depends on providers and peering we cannot see from here. It will not tell you what your licence charges, because that is a commercial negotiation and the structures vary; the terms piece covers what those structures look like. It will not tell you whether you may cache, replicate or train on what crosses the border — that turns on provenance and permission, treated in the licensing guide and, on the question of building the corpus yourself, in the piece on what the source portal permits.

Nor does a good latency number make a corpus useful. A fast path to a corpus that does not contain what your users ask for is a fast path to nothing, which is why coverage verification comes first in the evaluation order set out in the verification protocol, and why the field-level questions in the piece on structuring PRC court data decide how much of the work sits on your side at all. Latency is the last constraint to optimise and the first one to disqualify a design: worth computing early, worth optimising late.