An expert AI data project turns specialist knowledge into material a model can learn from or be evaluated against. That might mean worked examples, comparisons between responses, or assessments against a domain-specific rubric.

The format matters, but the first question is more basic: what should your model do better? A useful brief connects the intended behaviour to the people, tasks, and review process that can help you measure it.

Start with one sentence. “We need [type of expert] to create or review [type of task] in [language and context], so we can assess or improve [specific model behaviour].”

01. Define the behaviour

Choose a specific use case. “Improve financial reasoning” leaves too much open. “Evaluate whether answers correctly interpret the assumptions in a French-language financial scenario” gives contributors and reviewers something concrete to work with.

Specify the intended user, the information available to the model, the expected output, and the kinds of mistake that matter. Keep tasks that will train a model separate from tasks used to evaluate the result, and agree how that separation will be maintained.

02. Describe the expertise

Define the knowledge needed to do the task well. A job title alone may be too broad. Relevant practice, professional qualifications, language proficiency, and familiarity with a local context can each matter.

Separate essential requirements from preferences. Decide how qualifications will be checked and how a short calibration exercise will establish that contributors understand the work. Set payment terms for calibration and production explicitly.

03. Make the task concrete

Provide a small set of worked examples. Include a clear success, a clear failure, and an ambiguous case that needs explanation. State what contributors may use as supporting material and whether any AI assistance is permitted.

Define the output format. For example, a comparison task could contain two model responses, a preference decision, the reason for that decision, and the criteria used. Do not leave reviewers to infer the meaning of a field.

04. Agree what passes review

Write the rubric before production. Break “good” into criteria that a reviewer can apply: factual accuracy, relevance, completeness, reasoning, or appropriate handling of uncertainty.

Use a sample to find unclear instructions and disagreements. Agree who resolves difficult cases, what gets reworked, and how acceptance is decided. Set any agreement target after examining the task; one universal threshold will not suit every project.

05. Set the data boundaries

Record where source material comes from, what contributors may access, and what rights the project needs. Identify any sensitive material before it enters the workflow.

Agree storage, permitted tools, access, retention, deletion, and delivery requirements. Specify whether the resulting dataset is exclusive and whether any reuse is allowed. Those are project terms to resolve explicitly, not assumptions to carry into production.

06. Scope a pilot you can judge

Choose a bounded task set, timetable, and budget. Define the accepted output and the review evidence you need alongside it. Make the decision after the pilot clear: revise the brief, expand the work, or stop.

A useful pilot produces more than files. It shows whether the instructions work, where experts disagree, and what changes are needed before a larger commitment.

Start with a short overview. Tell us your use case and the expertise you need. We can work through the rest of the brief together.

Tell us about your project

Bring the brief to the conversation

Legendre’s proposed starting point is a focused expert data project, with requirements and review criteria agreed before production. If you have a use case in mind, that is enough to start discussing the scope.

Discuss a project

Further reading

The InstructGPT paper describes a workflow using human demonstrations and comparisons. Datasheets for Datasets proposes documenting dataset motivation, collection, composition, and intended uses. The checklist above is Legendre’s proposed project-scoping approach.