Skip to main content

Evaluator templates

Evaluator templates let you save a fully configured evaluator and reuse it across test runs in your project. Templates are stored at the project level.

Why use templates​

  • Standardize quality criteria across teams or projects.
  • Stop re-authoring the same regex / schema / LLM judge prompt for every test run.
  • Bootstrap new evaluators by importing a template and tweaking.

Save an evaluator as a template​

From any evaluator configured inline in a test run, click Save as template.

  • Name and description aren't asked again — they're taken from the evaluator config you already filled out (evaluator name → template name; evaluator description → template description).
  • The template is stored at the project level and becomes available to all members with project access.

Template types​

TypeEditable?Deletable?
User templatesYesYes
System templatesNoNo

System templates are platform-provided and cover common scoring patterns out of the box.

System templates included​

  • LLM-as-a-Judge — the standard "new evaluator" templates from the previous evaluator flow.
  • Python Executor — at minimum: regex matching, JSON schema matching, keyword matching.

You can import a system template and customize it, but the underlying system template itself remains unchanged.

Project-level template management​

A dedicated Evaluator templates screen in the Evaluations section lets you:

  • View all templates available to the project (user + system).
  • Create new templates directly without going through a test run.
  • Edit user templates (system templates are read-only).
  • Delete user templates.

When editing or creating a Python evaluator template on this screen, optionally select an application to populate dataset, run_output, and trace sample payloads in the editor. If no application is selected, sample payloads aren't shown.

Import a template into a test run​

From the test run evaluator setup, click Import template and pick a template. The template's configuration prefills the evaluator. From there:

  • Edit any field — the change applies only to this test run, not to the source template.
  • Save back as a new template if your changes are reusable.

Audit log​

Template lifecycle events are captured in the audit log:

  • Template created
  • Template edited
  • Template deleted

Resource type: Evaluator template.

Versioning of templates​

Templates themselves are not versioned today — editing a user template overwrites the previous configuration. To preserve old configurations, save under a new name before making changes.