Skip to main content

Evaluators

Evaluators score each row of a test run against criteria you define. SimplAI supports two evaluator types: LLM-as-a-Judge and Python Executor.

You can add multiple evaluators to a test run, and each one is configured inline — there is no separate "evaluator resource" to create and no variable-mapping UI to fill out.

Common evaluator fields​

Every evaluator, regardless of type, requires:

  • Evaluator name (mandatory). Evaluator names must be unique within a test run — this matters for multi-version comparison eligibility and for matching evaluators across the comparison table.
  • Evaluator description (optional). Helpful context shown next to the evaluator in the test run UI and used by Autofill with AI.

When you save an evaluator as a template (see Evaluator templates), the name and description are not asked again — they're taken from the evaluator config you already filled out.

LLM-as-a-Judge​

Use an LLM as a judge to score outputs against natural-language criteria.

Fields​

  • Model — pick the model connection used to run the judge.
  • Prompt — a Jinja-templated prompt that produces the judgment. Supports full Jinja templating: filters, conditionals, formatting.
  • Scoring format — one of:
    • Numeric — set min and max values.
    • Categorical — define labels and mark the best / most desired label. The first label is the default best.
    • Binary — true/false (pass/fail equivalent).

Prompt templating​

In the prompt, reference:

  • Dataset fields — values from the dataset row being evaluated.
  • Run outputs — what the application produced for that row.
  • Traces — the run's trace payload (when trace availability is enabled for evaluations).

You can use nested access and lightweight manipulation via Jinja filters and conditionals.

Example: Numeric LLM judge​

You are evaluating customer service quality on a scale of 1–5.

User question: {{dataset.question}}
Agent response: {{run_output.reply}}

Score the response on:
- Politeness
- Accuracy
- Conciseness

Return a single integer between 1 and 5.

Scoring format: Numeric, min 1, max 5.

Python Executor​

Use a Python executor when your criteria are deterministic or need precise control (regex matching, JSON schema validation, keyword presence, custom logic).

Fields​

  • Scoring format — Numeric, Categorical, or Binary (same options as LLM judge).
  • Code editor — implement the __main__ function.

Function contract​

def __main__(dataset, run_output, trace):
    # dataset:    dict — one dataset row
    # run_output: dict — that row's application output
    # trace:      dict — that row's trace payload
    # Return:
    #   Numeric format     → a number within the declared min/max
    #   Categorical format → one of the declared labels (string)
    #   Binary format      → True or False
    ...

Sample payloads​

Below the code editor, the platform shows sample payloads so you can author the function against real shapes:

  • dataset sample.
  • run_output sample.
  • trace sample.

Sample rules:

  • Agents — standardized schemas shown accurately.
  • Tools — inferred from the tool input/output schema; the trace sample covers all unique tool steps that can occur.

Example: keyword matching​

def __main__(dataset, run_output, trace):
    required = ["refund", "policy"]
    text = (run_output.get("reply") or "").lower()
    return all(word in text for word in required)

Scoring format: Binary.

Note on the trace argument​

The trace argument is available when trace-availability for evaluations is enabled in your environment. If not yet enabled, the trace argument is empty — author evaluators against dataset and run_output only.

Save as template​

After configuring an evaluator, click Save as template to make it reusable across the project. See Evaluator templates for the template lifecycle.

Pricing​

  • LLM judges are billed per evaluation invocation + shared model token charges (when using the default platform models).
  • Python evaluators are billed per second of execution time at the standard Python code step rate.

See Overview for the full cost model and how charges are surfaced in usage line items.