Build an evaluation set before changing your prompt

Illustrative editorial sample. This draft describes a general workflow, not a current announcement, quotation or measured benchmark. It requires editorial review before publication.

A practical starting point

An illustrative method for comparing changes on the tasks your product actually supports.

1. Define the boundary

Collect representative tasks and document what a useful result means for each. Include ambiguous and failure cases.

2. Inspect the evidence

Keep a holdout set separate from examples used during development. Record dataset changes so comparisons remain interpretable.

3. Make the decision explicit

Combine automated checks with human review where judgment matters. A score is meaningful only in the context of its rubric.

4. Test the failure path

Compare the same inputs, settings and acceptance criteria. Report limitations and avoid claiming that a local result proves general capability.

Review before use

Adapt this checklist to your application, record the assumptions and verify any product-specific facts against primary sources. This sample does not provide a news report, legal opinion or performance guarantee.