---
title: "Test an AI workflow before you depend on it"
description: "Build a small test set, define useful success criteria, and measure the corrections as well as the output."
author: Graham
date: 2026-09-06
url: https://powerofai.ca/learn/test-an-ai-workflow
---

# Test an AI workflow before you depend on it

Build a small test set, define useful success criteria, and measure the corrections as well as the output.

A convincing demo tells you what worked once. A test set tells you where it breaks.

## Define success in observable terms

For an action-item extractor, success might mean each item has a supported action, an owner when one is stated, and no invented deadline. For a draft reply, it might mean the policy is accurate and the next step is clear. “High quality” is too vague to score consistently. Decide the criteria before looking at a new result so your standard does not move with the answer.

## Include normal, awkward, and missing-input cases

A small starting set could include a straightforward note, a note with two people of the same name, a cancelled action, a missing deadline, and an empty file. Add examples of failures you have actually seen. Keep some inputs aside while revising the prompt, then test those after the revision. Repeatedly tuning on the same examples can hide how poorly the workflow handles new material.

## Change one thing and compare

Run the old and new approach on the same cases under comparable conditions. Record the tool and relevant settings when possible. Score against the criteria, not against which version sounds more polished. If outputs vary, repeat important cases. A small test set can reveal defects, but it does not establish universal reliability or a market-leading benchmark.

## Count the work after generation

Measure preparation, waiting, review, and corrections. An assistant that drafts in one minute but requires 25 minutes of checking may not help a 15-minute task. Keep a simple log of elapsed time and failure types. Use the workflow calculator to explore your own estimates, then replace those estimates with observed times before making a purchasing or automation decision.

## A worked example

Original teaching example, not a recorded model run.

Vague starting point: This worked on my example. Automate it.

### A more useful brief

Test this extraction on five cases: ordinary, missing owner, cancelled task, conflicting dates, and empty input. Mark each criterion pass or fail and show supporting text. Do not invent missing values. List failures before suggesting changes.

Why this helps: The test challenges the failure modes rather than replaying only the easiest example.

## Your turn

Create five cases for a recurring task and write three pass/fail criteria. Try the workflow and record one actual failure.

You are looking for this: You can describe what failed, on which input, and what change you will test next.

## A quick check

You improve a prompt using five examples. How do you check that it generalizes?

1. Only rerun the easiest example
2. Judge how confident the answer sounds
3. Try held-aside examples and relevant edge cases

Answer: Try held-aside examples and relevant edge cases

Fresh cases help reveal whether the improvement extends beyond the examples used to tune it.

## Sources and how to read this guide

Examples and exercises are original teaching material, not recorded model runs, measured time savings, or certifications. Official sources were consulted September 6, 2026. Product behavior can change.

- [OpenAI · Prompt engineering](https://developers.openai.com/api/docs/guides/prompt-engineering): Official background on instructions, context, examples, and evaluating changes.
- [Anthropic · Building effective agents](https://www.anthropic.com/engineering/building-effective-agents): Distinguishes predefined workflows from systems that choose their own steps.

## Take the idea further

- [Compare AI tools on your own work](/learn/compare-ai-tools)
- [Chat, workflow, or agent: choose how much work to delegate](/learn/from-chat-to-agent)

[All learning paths](/learn) | [Practice challenges](/practice).