An expert AI data project turns specialist knowledge into material a model can learn from or be evaluated against. That might mean worked examples, comparisons between responses, or assessments against a domain-specific rubric.

The format matters, but the first question is more basic: what should your model do better? A useful brief connects the intended behaviour to the people, tasks, and review process that can help you measure it.

Start with one sentence. “We need [type of expert] to create or review [type of task] in [language and context], so we can assess or improve [specific model behaviour].”

01. Define the behaviour

Choose a specific use case. “Improve financial reasoning” leaves too much open. “Evaluate whether answers correctly interpret the assumptions in a French-language financial scenario” gives contributors and reviewers something concrete to work with.

Specify the intended user, the information available to the model, the expected output, and the kinds of mistake that matter. Keep tasks that will train a model separate from tasks used to evaluate the result, and agree how that separation will be maintained.

02. Describe the expertise

Define the knowledge needed to do the task well. A job title alone may be too broad. Relevant practice, professional qualifications, language proficiency, and familiarity with a local context can each matter.

Separate essential requirements from preferences. Decide how qualifications will be checked and how a short calibration exercise will establish that contributors understand the work. Set payment terms for calibration and production explicitly.

03. Make the task concrete

Provide a small set of worked examples. Include a clear success, a clear failure, and an ambiguous case that needs explanation. State what contributors may use as supporting material and whether any AI assistance is permitted.

Define the output format. For example, a comparison task could contain two model responses, a preference decision, the reason for that decision, and the criteria used. Do not leave reviewers to infer the meaning of a field.

04. Agree what passes review

Write the rubric before production. Break “good” into criteria that a reviewer can apply: factual accuracy, relevance, completeness, reasoning, or appropriate handling of uncertainty.

Use a sample to find unclear instructions and disagreements. Agree who resolves difficult cases, what gets reworked, and how acceptance is decided. Set any agreement target after examining the task; one universal threshold will not suit every project.

05. Set the data boundaries

Record where source material comes from, what contributors may access, and what rights the project needs. Identify any sensitive material before it enters the workflow.

Agree storage, permitted tools, access, retention, deletion, and delivery requirements. Specify whether the resulting dataset is exclusive and whether any reuse is allowed. Those are project terms to resolve explicitly, not assumptions to carry into production.

06. Scope a pilot you can judge

Choose a bounded task set, timetable, and budget. Define the accepted output and the review evidence you need alongside it. Make the decision after the pilot clear: revise the brief, expand the work, or stop.

A useful pilot produces more than files. It shows whether the instructions work, where experts disagree, and what changes are needed before a larger commitment.

A worked example: evaluate a support answer

Illustrative brief · Fictional product and outputs

A software team wants to evaluate whether its French-language assistant explains account permissions accurately. This example shows the decisions a buyer and supplier could agree. It is not a client case study or a universal evaluation standard.

Purpose
Evaluate answers against an approved product reference. Keep the evaluation set separate from material used to train or tune the assistant.
Expert profile
French-language software-support experience, clear written reasoning and successful calibration on the supplied reference.
Inputs
A versioned reference, a user question and the assistant’s answer. Use fictional accounts; no customer records are needed for this example.
Output
A judgment for each criterion, supporting evidence, uncertainty notes and a final review status.
Pilot decision
Review error patterns and unresolved disagreements before commissioning more tasks. Accepting the dataset does not by itself demonstrate a model improvement.

The task packet

Reference, version 1.0: « Les administrateurs peuvent exporter les rapports au format CSV. Seul le propriétaire du compte peut modifier la facturation. »

Question: « Je suis administrateur. Puis-je exporter les rapports et modifier la facturation ? »

Answer to review: « Oui, les administrateurs peuvent exporter les rapports et modifier la facturation. »

The answer gets the export permission right but assigns billing permission to the wrong role. A reviewer should identify that specific error and cite the reference.

A rubric with observable decisions

How the example answer is judged
CriterionPass conditionExample result
Reference accuracyEvery stated permission agrees with the supplied reference.Fail. Billing changes are limited to the account owner.
CoverageThe answer addresses both export and billing.Pass. Both are addressed, though one answer is wrong.
ClarityThe French is understandable and the roles are named explicitly.Pass. Fluent wording does not override the accuracy failure.

In this illustrative rubric, reference accuracy is required for an overall pass. The assistant’s answer therefore fails. The evaluation record can still pass data review if its judgments and evidence are correct. Agree this distinction before paying for accepted records.

Decide what happens when the source is silent

If a question asks about a permission the reference does not cover, use a separate “insufficient evidence” outcome. Reviewers should flag the gap rather than invent a rule. If reviewers disagree, record their reasons and route the case to the agreed adjudicator. Update the rubric version when a clarification changes the rules, and decide which earlier records need another review.

Make the delivery easy to inspect

For this example, one record could carry the fields below. Agree your schema and allowed values with the team that will use the data.

{
  "task_id": "sample-fr-001",
  "reference_version": "1.0",
  "rubric_version": "1.0",
  "criterion_results": {
    "reference_accuracy": "fail",
    "coverage": "pass",
    "clarity": "pass"
  },
  "answer_result": "fail",
  "evidence": "Billing changes require the account owner.",
  "data_review_status": "accepted"
}

Include the referenced task inputs, versioned instructions and a record of exclusions or rework with the delivery. Use project-specific reviewer IDs where traceability is needed, and agree who may access the underlying identities. A valid file format is only one part of acceptance.

Before production: a pilot checklist

  • Purpose: one behaviour to assess, an intended use and a named decision owner.
  • Coverage: agreed languages, task types and edge cases; a reason for the pilot size.
  • Calibration: examples of a pass, a failure and insufficient evidence, with payment terms for calibration work.
  • Review: acceptance criteria, review coverage, disagreement handling and rework responsibilities.
  • Data terms: approved sources, permitted tools, access, retention and rights to use the output.
  • Delivery: a sample file that the buyer can open, plus instructions, versions and review evidence.
  • Decision: a budget, timetable and an explicit choice to expand, revise or stop.

You do not need every answer before contacting a supplier. Separate decisions already made from questions that need scoping. Do not send confidential source data through a general enquiry form.

Start with your use case.

Tell us what your model should do better, which expertise or languages matter, and what kind of pilot you have in mind. A few sentences are enough to start.

Discuss your data pilot

Further reading

The InstructGPT paper describes a workflow using human demonstrations and comparisons. Datasheets for Datasets proposes documenting dataset motivation, collection, composition, and intended uses. The checklist above is Legendre’s proposed project-scoping approach.