# Build an evaluation set before changing your prompt

Illustrative editorial sample. This draft describes a general workflow, not a current announcement, quotation or measured benchmark. It requires editorial review before publication.

## A practical starting point

An illustrative method for comparing changes on the tasks your product actually supports.

## 1. Define the boundary

Collect representative tasks and document what a useful result means for each. Include ambiguous and failure cases.

## 2. Inspect the evidence

Keep a holdout set separate from examples used during development. Record dataset changes so comparisons remain interpretable.

## 3. Make the decision explicit

Combine automated checks with human review where judgment matters. A score is meaningful only in the context of its rubric.

## 4. Test the failure path

Compare the same inputs, settings and acceptance criteria. Report limitations and avoid claiming that a local result proves general capability.

## Review before use

Adapt this checklist to your application, record the assumptions and verify any product-specific facts against primary sources. This sample does not provide a news report, legal opinion or performance guarantee.


An illustrative method for comparing changes on the tasks your product actually supports.

Build an evaluation set before changing your prompt

Sample Editorial Desk

# Designing an agent that knows when to ask

Illustrative editorial sample. This draft describes a general workflow, not a current announcement, quotation or measured benchmark. It requires editorial review before publication.

## A practical starting point

An illustrative approval workflow for tools that change external systems.

## 1. Define the boundary

Start with a narrow task and an explicit list of tools. Give the agent the minimum access required for that task.

## 2. Inspect the evidence

Separate a proposed action from its execution. Show the exact target, changes and consequences before approval.

## 3. Make the decision explicit

Bind approval to a specific request and revision. If the resource changes during review, inspect it again.

## 4. Test the failure path

Keep a useful event trail: requested action, tool response, approval and result. Test cancellation and partial failure as carefully as success.

## Review before use

Adapt this checklist to your application, record the assumptions and verify any product-specific facts against primary sources. This sample does not provide a news report, legal opinion or performance guarantee.


An illustrative approval workflow for tools that change external systems.

Designing an agent that knows when to ask

# Treat retrieved instructions as untrusted content

Illustrative editorial sample. This draft describes a general workflow, not a current announcement, quotation or measured benchmark. It requires editorial review before publication.

## A practical starting point

An illustrative security review for systems that read documents and use tools.

## 1. Define the boundary

Identify the trust boundary between application instructions and retrieved material. A document should not grant new permissions.

## 2. Inspect the evidence

Authorize each tool call using application identity and policy. Do not rely on the model to enforce access control.

## 3. Make the decision explicit

Redact secrets before context construction. Include only resources the current user is permitted to inspect.

## 4. Test the failure path

Test hostile text, misleading links, malformed outputs and attempts to access another account. Verify rejection without leaking private context.

## Review before use

Adapt this checklist to your application, record the assumptions and verify any product-specific facts against primary sources. This sample does not provide a news report, legal opinion or performance guarantee.


An illustrative security review for systems that read documents and use tools.

Treat retrieved instructions as untrusted content

# Design retrieval around evidence, not more context

Illustrative editorial sample. This draft describes a general workflow, not a current announcement, quotation or measured benchmark. It requires editorial review before publication.

## A practical starting point

An illustrative retrieval workflow with clear source ownership and failure handling.

## 1. Define the boundary

Define the question types the collection is meant to answer. Track document origin, access rights and update time.

## 2. Inspect the evidence

Choose chunking and retrieval settings using representative queries. Inspect the actual retrieved passages instead of only the final response.

## 3. Make the decision explicit

Link answers to evidence that supports the claim. If a passage does not answer the question, allow the system to say so.

## 4. Test the failure path

Test stale, contradictory and inaccessible documents. Enforce document permissions before retrieval and before displaying citations.

## Review before use

Adapt this checklist to your application, record the assumptions and verify any product-specific facts against primary sources. This sample does not provide a news report, legal opinion or performance guarantee.


An illustrative retrieval workflow with clear source ownership and failure handling.

Design retrieval around evidence, not more context

# Before the headline: a checklist for verifying AI announcements

Illustrative editorial sample. This draft describes a general workflow, not a current announcement, quotation or measured benchmark. It requires editorial review before publication.

## A practical starting point

Separate a primary announcement from commentary before drafting a news story.

## 1. Define the boundary

Find the dated announcement on the organization’s own website. Record both the event date and the date you checked it.

## 2. Inspect the evidence

Compare the announcement with the release notes, model card or technical report it references. Mark missing details rather than filling them in.

## 3. Make the decision explicit

Distinguish availability, demonstrations and measured capability. A planned release is not a deployed product.

## 4. Test the failure path

Keep quotations exact and attributed. Do not turn marketing language into an independent benchmark claim.

## Review before use

Adapt this checklist to your application, record the assumptions and verify any product-specific facts against primary sources. This sample does not provide a news report, legal opinion or performance guarantee.


Separate a primary announcement from commentary before drafting a news story.

Before the headline: a checklist for verifying AI announcements

# Build a model API boundary you can change later

Illustrative editorial sample. This draft describes a general workflow, not a current announcement, quotation or measured benchmark. It requires editorial review before publication.

## A practical starting point

An illustrative integration checklist that keeps provider details out of product flows.

## 1. Define the boundary

Put provider requests behind a small server-side interface. Keep API credentials out of browser code, logs and sample configuration.

## 2. Inspect the evidence

Validate input and output against the product’s own contract. Treat generated output as data that may be incomplete or unexpected.

## 3. Make the decision explicit

Set bounded timeouts and cancellation. Decide which requests may be retried safely, and show an honest failure state.

## 4. Test the failure path

Measure latency and cost on representative tasks before choosing defaults. Keep a fallback policy explicit and test it with recorded fixtures.

## Review before use

Adapt this checklist to your application, record the assumptions and verify any product-specific facts against primary sources. This sample does not provide a news report, legal opinion or performance guarantee.


An illustrative integration checklist that keeps provider details out of product flows.

Build a model API boundary you can change later

# Evaluate image-and-text workflows as a whole

Illustrative editorial sample. This draft describes a general workflow, not a current announcement, quotation or measured benchmark. It requires editorial review before publication.

## A practical starting point

An illustrative checklist for products that combine visual inputs and written responses.

## 1. Define the boundary

Describe the supported input types and quality limits. Tell users when a photo or scan is too unclear for a reliable result.

## 2. Inspect the evidence

Include accessible alternatives and preserve meaningful image descriptions. Obtain the permissions needed to process uploaded material.

## 3. Make the decision explicit

Evaluate the complete workflow, including upload validation, context construction and the displayed answer.

## 4. Test the failure path

Test ambiguous imagery and missing context. Require a human decision where the result affects a consequential action.

## Review before use

Adapt this checklist to your application, record the assumptions and verify any product-specific facts against primary sources. This sample does not provide a news report, legal opinion or performance guarantee.


An illustrative checklist for products that combine visual inputs and written responses.

Evaluate image-and-text workflows as a whole

Editorial perspectives on Model Evaluation.

Build your next website

Website templates

Launch your online store

Ecommerce

A shared language for people and agents

Build with SigmaUI

Find your starting point

Learn and explore

Resources

Model Evaluation

Articles

Build an evaluation set before changing your prompt