Skip to main content

Multi-version comparison

Run the same dataset and evaluators across up to 6 versions of an application — side by side. The platform identifies the best version per evaluator, color-codes the cells, and lets you bring prior runs into the comparison without re-evaluating.

Selecting versions​

In the evaluation setup screen:

  • Pick a primary version + up to 5 comparison versions (max 6 total).
  • Click Add version to compare to add additional dropdowns up to the cap.
  • Each dropdown only shows versions not already selected in another dropdown.
  • Comparison versions can be removed. The primary version cannot.
  • If you hover over Add version to compare at the limit, a tooltip explains: max 6 versions.

The dataset selection is unchanged — the same dataset applies to all selected versions.

Execution model​

When you run the evaluation with multiple versions selected:

  • Each version produces a separate test run row in the test runs table.
  • All those rows are linked as a comparison set for the view-and-analyze flows below.

What you see in the results page​

If you open a test run that belongs to a comparison set, a comparison table appears above the standard detailed results.

Aggregated comparison table​

  • Column 1 — Evaluator name.
  • Column 2 — Metric value for the version you opened.
  • Columns 3..N — Metric values for the other versions in the set.

For each evaluator (each row):

  • The best value is bold with a clear indicator.
  • Cells use color scaling to show best / middle / worst at a glance.

The aggregated metric depends on the evaluator's scoring format (same as the standard results page):

  • Numeric — average score.
  • Categorical — count of the best label.
  • Binary — count of true.

Detailed side-by-side table​

Below the comparison summary, the detailed table shows row-level comparison:

  • Inputs section — one row per dataset row, with Input message/Inputs, Expected response, start time.
  • Outputs section — outputs from all versions side by side. Per version: agent response (or Tool output / chain reply), Latency, Run Status, Trace ID.
  • Evaluator results section — single Evaluator-name column; per version: Score/Label, Explanation, Evaluator status.

If multiple evaluators are configured, the evaluator results section splits into sub-rows per evaluator. Example: 10 dataset rows × 4 evaluators = 40 evaluator sub-rows (4 per dataset row).

Add prior runs to a comparison​

You can pull a previously executed test run into the current comparison view — no re-evaluation needed.

  • Available up to the total cap of 6 versions in the comparison.
  • You can remove comparison versions, except the version whose results are currently open.

Eligibility​

A prior run can be added to the comparison only if it matches the current run on all of:

  • Application
  • Dataset
  • Set of evaluator names — the names must match exactly as a set (order-independent). Evaluator configuration may differ; the eligibility check is name-based.

This is why evaluator names must be unique within a run — see Evaluators for the naming constraint.

Adding or removing reflects instantly — no new compute, no waiting.

Run on the latest version​

A Run on <latest_version_name> button is shown above the comparison view when both conditions hold:

  • The comparison set has fewer than 6 versions.
  • The latest named version of the application is not already included.

If the button is disabled (limit reached, or latest is already in the set), hover to see a tooltip explaining why.

While the latest run is processing​

  • Columns for the latest version appear immediately with wireframe loaders for output and evaluator results.
  • The in-progress run shows up as a separate row in the test runs table.
  • When the run completes, loaders resolve automatically to the actual values — no manual refresh needed.

Usage billing​

Each version-run is billed as a separate test run (consistent with how runs appear in the test runs table). The comparison view itself is free — it's view-only and adds no compute. Run on latest version bills as one additional run.

In the audit log​

Multi-version evaluation events are captured in the audit log:

  • Test run created — including version info and whether the run is part of a multi-version set.
  • Run on latest version triggered.