原文核对：2026-10-13 · 中文正文 · 英文提示词与演示
场景定位
检查 AI 应用，明确待评估行为，并建立可运行的 Promptfoo 评测集。
- 难度
- 中级
- 时间跨度
- 1 小时
适合用于
- 已有提示词、模型、工具、检索或代理，但缺少重复评测的应用。
- 模型、提示词、检索或代理变更合并前需要回归测试的团队。
- 希望将人工质量检查沉淀为仓库评测案例的团队。
相关工具
提供 $promptfoo-evals 与 $promptfoo-provider-setup，用于创建、连接、运行和检查评测。
Promptfoo ↗起步提示词
英文原文 · 可复制后按实际材料修改
Use $promptfoo-evals to add a Promptfoo eval suite for this AI application. If there is not already a working Promptfoo provider or target adapter, use $promptfoo-provider-setup first.
Behavior to evaluate: [support answer quality / tool-call correctness / retrieval grounding / business rules / agent task completion]
Before editing:
- Inspect the app path users hit and any existing evals or tests.
- Propose the smallest useful eval plan: target adapter, seed cases, assertions, files, commands, and required env vars or local services.
- Do not change production prompts, model settings, or app behavior until the baseline eval exists and has been run.
Requirements:
- Exercise the application path users hit when possible, not only the raw model prompt.
- Keep fixtures free of secrets, customer data, and sensitive personal data.
- Add a local eval command such as `npm run evals` or document the exact command to run.
Finish with:
- Files changed
- Eval commands run
- Passing and failing cases
- Recommended next evals to add简介
评测系统地测试一组情景，在发布前发现应用行为回归。
用 Promptfoo 运行评测，用 Codex 创建和维护。
使用方法
通过 $promptfoo-evals 将一项行为转成可重复评测；没有可用目标时，用 $promptfoo-provider-setup 接入实际应用路径。
Codex 可检查应用、提出有效案例、加入配置与数据、本地运行，并留下可复用命令。
适合客服回答质量、检索依据、分类标签、工具调用、JSON 格式、业务规则或提示词与模型迁移等具体行为。
首版应交付可审阅代码和测试数据：promptfooconfig.yaml 或等效配置、小型 evals/、案例、必要适配器及 npm run evals 等命令。
选择评估内容
从一个用户可见承诺开始，不要一次评估整个系统；小型套件更易信任、审阅和持续运行。
首批目标可包括：
- 正确性：分类、提取、摘要、路由或转换。
- 依据：回答保持与检索文档或引用来源一致。
- 工具：正确选择、有效参数和错误处理。
- 格式与业务规则：JSON schema、字段名、业务边界或 UI 文案约定。
- 迁移：新提示词、模型、系统消息或检索设置不破坏关键案例。
从产品要求、缺陷、客服升级记录或允许提交仓库的脱敏示例出发。
先制定评测计划
修改前检查应用，列明目标路径、fixture、断言、适配器和命令，以便发现错误目标或薄弱案例。
实施前审阅调用路径/端点、种子案例、断言、新文件、命令及所需密钥或服务。若测的是裸模型而非用户实际应用路径，应确认是否有意如此。
实现、运行和迭代
计划正确后实现配置、案例、fixture、必要适配器和命令，并提供实际运行证据。
小型应用评测集可以如下组织：
evals/
promptfooconfig.yaml
tests/
cases.yaml
providers/
provider.js # only if the built-in provider cannot call the app directly改行为前先跑基线，判断已有产品失败、断言问题或适配器错误。修正过脆弱或模糊的断言，但保留真实产品失败。
随后用评测比较发布前变更，按缺陷、上线要求和产品审阅补案例；本地命令稳定后接入 CI 或发布清单。
展开英文原文对照
Ask Codex to inspect your AI application, identify the behavior you want to evaluate, and add a runnable Promptfoo eval suite.
Introduction
When you are building an AI application, or making changes to an existing one, you want to make sure it behaves as expected. Evals are a way to systematically test a set of scenarios and catch regressions before they ship.
You can use Promptfoo to run evals on your AI application, and Codex to help you create and maintain the evals.
How to use
Use Codex with the Promptfoo plugin’s $promptfoo-evals skill to turn one AI app behavior into a repeatable eval suite. When the app does not already have a working Promptfoo target, $promptfoo-provider-setup helps connect the suite to the application path you want to test.
Codex can inspect the app, propose high-signal cases, add the Promptfoo config and test data, run the suite locally, and give you a command to keep using.
This use case works best when the behavior is concrete: support answer quality, retrieval grounding, classifier labels, tool calls, JSON shape, business rules, or prompt and model migration confidence.
A strong first pass should be reviewable code and test data: a promptfooconfig.yaml or equivalent config, a small evals/ directory, test cases, any target adapter needed to call the app, and a local command such as npm run evals.
Choose what to evaluate
Start with one user-visible promise. Avoid asking Codex to evaluate the entire AI system in one pass. A smaller suite is easier to trust, review, and keep running.
Good first targets include:
- Correctness: classification, extraction, summarization, routing, or transformation.
- Grounding: answers that should stay tied to retrieved documents or cited sources.
- Tool use: choosing the right tool, passing valid arguments, and handling tool errors.
- Format or business rules: JSON schemas, field names, business-rule limits, or UI-facing copy contracts.
- Prompt or model migration: making sure a new prompt, model, system message, or retrieval setting does not break important cases.
Start from product requirements, bug reports, support escalations, or sanitized examples your team is comfortable committing to the repo.
Ask for an eval plan
Codex should inspect before it edits. Ask for a plan that names the target path, fixtures, assertions, adapter, and commands. This gives you a chance to catch the wrong target or weak test cases before files are added.
Review the plan before implementation. It should name the app path or endpoint Promptfoo will call, the first seed cases, the assertions, the files Codex will create, the local command, and any required secrets or services. If the plan tests the raw model instead of the application path users hit, ask Codex whether that is intentional.
Implement, run, and iterate
Once the plan is correct, ask Codex to implement it. The first implementation should be boring: config, cases, fixtures, a target adapter if needed, a command, and proof that the command ran.
A small app-backed suite might look like this:
evals/
promptfooconfig.yaml
tests/
cases.yaml
providers/
provider.js # only if the built-in provider cannot call the app directlyRun the suite before changing behavior. The baseline tells you whether the app already fails the cases, whether the assertions need tuning, or whether the target adapter is wrong. Tune assertions when they are too brittle or vague, but keep real product failures visible.
After the first run, use the suite to compare app changes before they ship. Add new cases whenever a bug, launch requirement, or product review shows behavior you want to keep stable. Once the local command is stable, ask Codex to add it to CI or your release checklist.