Who Owns the Artefacts of an Evaluation? Seed Lists, Eval Sets and Field Mappings in a Chinese Legal Data Deal

Every data negotiation spends its energy on one ownership question: who owns the data. It is the wrong one to spend it all on, because the licence answers it, the answer is rarely surprising, and it is not the thing that determines your position eighteen months later.

The question almost nobody scopes is who owns the things you made while testing the data. Your team writes a list of matters it must be able to find, adjudicates which results are actually relevant, maps a supplier's fields onto your own model, writes the rules that reconcile court names and dates, builds a harness, and logs everything that came back wrong. None of that is delivered by the supplier. All of it is expensive, most of it is reusable, and its ownership is usually governed by whatever the pilot paper happened to say about material created during the engagement — frequently by nothing considered.

The piece on pilot design lists artefact ownership as one of seven clauses that belong in a pilot agreement and gives it four sentences. This page is those four sentences expanded — the clause is more contested than its length suggested, and the reliable fix turns out not to be a clause at all.

What an evaluation actually produces

Name the outputs before arguing about them. A properly run evaluation of a Chinese case law corpus leaves behind six artefacts.

Every item in that right-hand column is worth more at the end of the evaluation than the trial data itself, because the data expires with the trial and the artefacts do not. That is the whole argument for spending an hour on the clause.

Three mechanisms that capture them without anyone deciding to

Artefact ownership is more often captured by general wording drafted for something else than by a supplier reaching for your work.

1. The definition of derivative material

Data licences define a category of material created from the licensed corpus and then restrict what may be done with it. The definitions vary enormously in reach. A narrow one covers material that reproduces licensed content — extracts, transformed copies, embeddings of the text. A broad one covers material created by reference to the corpus, wide enough to include an evaluation set built out of the corpus's own record identifiers and an annotation file saying which of its documents were relevant. Nobody drafts that intending to take your grading work. It takes it anyway.

2. Work created during the engagement

A pilot paper or a professional-services template may assign, or jointly own, material developed in the course of the engagement. The template was written for a consulting deliverable, where joint ownership is a sensible default. Applied to a data pilot it can reach a field mapping written entirely by your own engineers, on the reasoning that it was developed during the engagement and describes the supplier's schema.

3. Confidentiality, which does not transfer ownership and does not need to

This is the one that catches sophisticated buyers. The agreement may leave your artefacts unambiguously yours while placing the entire engagement, including everything learned in it, under a confidentiality obligation with no carve-out for your own work product. You own the evaluation set. You may not show its results to anyone. For the purpose that made you build it, the distinction is academic.

Ownership and usability are separate fights, and the second one bites

The single most useful reframing here: settle title and settle disclosure as two questions, not one. A buyer can negotiate hard on the first, win it, and find at the next procurement round that the numbers cannot go in a comparison deck, the evaluation set cannot be handed to a second supplier for a like-for-like run, and the board cannot be told what the measured coverage was without a redaction pass.

Write both down. Title: the artefact is yours, in whole, on termination, without further licence. Disclosure: what you may say about the results, to whom, and whether the supplier may be identified when you say it. Those two sentences are cheap while the deal is being written and close to unobtainable afterwards, when the leverage has moved.

Make the artefact so that ownership is barely arguable

This part matters more than the drafting. Artefact ownership becomes genuinely contestable, as opposed to sloppily contestable, because teams build evaluation sets by pasting supplier content into them. A row carrying a paragraph of judgment text alongside its relevance grade is partly the supplier's licensed content. Asking to keep that file forever, after termination, is asking to retain a copy of licensed material, and a supplier who refuses is declining to hand over an uncontrolled extract of their corpus. On that framing they are right.

So do not build it that way. Store pointers, not payloads:

EVALUATION SET ROW  — pointers, not payloads

  query_id          your identifier, your wording
  query_text        yours
  target_ref        court case number  (a public docket identifier)
  supplier_ref      the supplier's internal id, in a side table
  grade             0-3, your rubric
  adjudicator       who graded it
  graded_on         date
  notes             your reasoning, in your words

  NOT IN THIS FILE: judgment text, headnotes, supplier-normalised
  fields, or any span copied from a delivered record.

Built this way, the artefact contains no licensed content. It contains your queries, your grades, your reasoning, and identifiers that exist independently of any supplier because a case number is assigned by a court. Keep the supplier's own internal identifiers in a separate mapping table you can drop on termination without damaging the artefact. The ownership question then has very little to bite on, and the same file runs against the next supplier on day one without a negotiation.

The same discipline applies to the defect log — record the identifier and the nature of the defect rather than pasting the malformed record — and to the harness, which should read supplier content at runtime and never persist it. The coverage verification protocol is built on this shape, and not by coincidence: a test designed for portability is also a test whose artefacts you can keep.

Four questions that settle the clause

Take these to whoever reads the paper. Each is answerable in a short exchange.

1. DERIVATIVE REACH
   Does the definition cover material that merely REFERENCES the
   corpus (identifiers, citations, relevance grades), or only
   material that REPRODUCES it (text spans, embeddings of text)?
   Ask for the second. Get the answer in the definition, not in
   an assurance.

2. CARVE-OUT
   Is buyer-created evaluation material named and excluded from
   both the derivative clause and any "developed during the
   engagement" clause? Name the artefacts. General language will
   be read against you later.

3. SURVIVAL, BOTH WAYS
   On termination: which artefacts do you keep, what must be
   deleted, and does confidentiality continue to attach to what
   you keep? An artefact you own under a perpetual confidentiality
   obligation is not portable.

4. DISCLOSURE RIGHT
   May you disclose measured results? To whom — internally,
   to a prospective supplier, publicly? Named or unnamed? If the
   answer is "with prior written consent", ask what happens when
   consent is withheld, because that is the operative term.

What a supplier can fairly ask for in return

A one-sided version of this page would be less useful. Several of the asks from the other side of the table are legitimate.

• Defects reported back. If your log identifies records that are missing or malformed, a supplier who wants to fix them needs to see it. Agreeing to share the defect log is cheap for you and is often the fastest route to the fix.
• Right of reply before a named public comparison. If you intend to publish a benchmark that names suppliers, letting them see the methodology and respond before publication is normal practice, and it protects you as much as them — measurement disputes are usually methodology disputes.
• No republication of record-level content. Publishing your results should not become a route to publishing their corpus. Reporting aggregate measurements rather than the documents behind them costs you nothing.
• Attribution of what was tested. A number measured on a two-week trial slice should not be reported as a property of the full corpus — an honesty point that happens to favour the supplier.

And the asks to decline: ownership or joint ownership of your field mapping; any claim over a seed list drawn from your own matters; a requirement that you obtain consent before showing unnamed results to another supplier; and confidentiality drafted so that your own measurements become the supplier's confidential information. The line runs between protecting a supplier from being misrepresented, which is reasonable in every deal, and preventing a buyer from comparing, which is what this page exists to stop.

Running two suppliers against one evaluation set

The artefact's highest use is comparison, and comparison means two agreements at once. Separate what is shared from what is not.

Shared, and safely so: your queries, your rubric, your seed list, your harness, and public docket identifiers. Not shared: each supplier's returned content, their internal identifiers, and — unless you have secured the disclosure right above — their results. The practical arrangement is one shared artefact plus a per-supplier result store keyed to the shared query identifiers, with supplier content held only for as long as the run needs it.

The failure mode is subtle and worth naming: a scoring pipeline that computes a head-to-head statistic needs both suppliers' outputs in one place at one time. That is a design decision with a disclosure consequence, made by an engineer, in a script, weeks after the lawyers finished. Compute per-supplier scores in isolation and compare the scores rather than the outputs — the same separation a multi-supplier stack needs anyway.

The portability test, before you sign

One exercise, fifteen minutes, and it converts an abstract clause into a concrete list. Write down what you would hand to a second supplier on the first day of a re-evaluation eighteen months from now. Be specific: the seed list file, the graded query set, the mapping document, the normalisation rules, the harness repository, the defect log, the summary of what the incumbent measured.

Now read the draft agreement with that list beside you and mark each line clearly mine, arguable, or clearly not mine. Anything in the middle column is a clause to fix now. The line most likely to land in the middle column is not the evaluation set — it is the results, caught by confidentiality rather than by ownership. That is the finding this test exists to produce.

Why the governing law is not boilerplate here

One reason this clause cannot be resolved from first principles: the background law is not the same everywhere, and cross-border data deals routinely put a corpus, a buyer and a governing-law clause in three different places.

In the European Union, the Database Directive creates a sui generis right for the maker of a database who has made a substantial investment in obtaining, verifying or presenting its contents — a right that exists independently of copyright and runs, under Article 10, for fifteen years from completion or first publication. In the United States there is no equivalent right: in Feist Publications v. Rural Telephone Service the Supreme Court rejected the “sweat of the brow” theory and held that a factual compilation attracts copyright only through originality in selection, coordination or arrangement. Those two regimes give materially different answers to the question of what a compiler can control.

The practical consequence is narrow: do not assume the default position on a compilation of facts — which is what an evaluation set largely is — matches the one you are used to. Write the ownership term explicitly rather than relying on what the governing law supplies, and have it read by someone qualified in that jurisdiction. This is background, not advice.

What this page does not settle

Several things, and they should be said plainly.

It does not tell you what your agreement says: a clause that looks like mechanism two may operate as mechanism one depending on definitions elsewhere in the document. Read yours.

It offers no data on how often these clauses are contested, because we have none that would survive scrutiny; nothing here is a frequency claim, and where one would have been natural it was left out on purpose.

And there is a disclosure to make rather than an objectivity to assert: we license data, so this page argues against the commercial interest of our own side of the table on several points. That is a position, not neutrality. Read the argument on its merits, take the parts that survive, and put the clause in front of your own counsel.