Skip to main content

Evaluation

The Evaluation surface lets you measure agent, tool, and chain quality against datasets or live traffic — and compare across versions to catch regressions before they ship.

This section covers:

  • Evaluators — LLM-as-a-Judge and Python executor evaluators, name/description, scoring formats (Numeric, Categorical, Binary).
  • Evaluator templates — save reusable evaluators at project level; user vs. system templates; import-and-edit workflow.
  • Autofill with AI — generate evaluator prompt or code + scoring from a name and description, using your dataset/output/trace schema.
  • Results table (Excel-like view) — aggregated cards, sortable spreadsheet-style results, filters, color-coded labels.
  • Multi-version comparison — compare up to 6 versions side-by-side; add prior runs; "Run on latest version" workflow.
  • Scheduled evaluations — cron-based runs over live runs, with sampling.
  • Exporting results — async CSV exports via email for offline analysis.

How evaluation runs are organized​

  • Every evaluation produces a test run — visible in the Evaluations tab as a row in the test runs table.
  • A test run includes a dataset (curated rows) or a sampled set of recent live runs, an application version (the Tool / Agent / Voice Agent under test), and one or more evaluators that score each row.
  • In multi-version comparison, each version selected produces a separate test run row, and the rows are linked as a comparison set for the side-by-side views.

Where evaluators run​

Evaluators run against:

  • Inputs, outputs, and (where available) traces for each row.
  • The Python evaluator function signature is __main__(dataset, run_output, trace) and returns the value matching its declared scoring format.

For voice agents, evaluation operates on conversation transcripts — see Voice Agent evaluation for the voice-specific notes (no dataset selection; full-transcript or per-turn input).

Cost model​

  • LLM-as-a-Judge evaluators are billed as a fixed charge per evaluation invocation + shared model token charges (when using the default platform models).
  • Python evaluators are billed per second of execution time at the standard Python code step rate.
  • For a single test application run, all LLM-judge units appear as one sub-line item (rows × evaluators), and all Python evaluator units appear as one sub-line item (total seconds).

Read order​

If you're new to Evaluation, read in this order:

  1. Evaluators — what an evaluator is and how to author one.
  2. Results table — what you'll see after a run.
  3. Multi-version comparison — how to compare versions.
  4. Scheduled evaluations — moving from one-off to continuous.