Skip to main content

Autofill with AI

Autofill with AI generates a working evaluator configuration from a short description, so you don't have to write the Jinja prompt or Python function from scratch.

Where it lives​

The Autofill with AI button appears in the evaluator configuration UI, below the evaluator description field. It's available on both evaluator types — LLM Judge and Python Executor.

What the autofill uses as input​

The autofill passes the following context to its underlying generator:

  • Evaluator name + evaluator description — what you typed.
  • Dataset details — column names and a sample row.
  • Tool / Agent input + output + trace schemas — so the generated prompt or code references the right field names.

You can optionally provide a custom prompt to steer the generation (e.g., "be lenient on minor formatting differences"). A default prompt is shown if you don't override it.

What gets generated​

For LLM-as-a-Judge​

The autofill produces:

  • A prompt (Jinja-templated, with references to dataset/run_output/trace fields).
  • A scoring format configuration — usually picked to match your description's intent (e.g., "fail on missing field" → Binary; "rate 1–5" → Numeric).

For Python Executor​

The autofill produces:

  • A __main__(dataset, run_output, trace) implementation ready to run.
  • A scoring format configuration.

Review before saving​

Autofilled evaluators are a starting point. Always:

  • Read the generated prompt or code — make sure it references the right fields.
  • Adjust the scoring format if it doesn't match what you actually want.
  • Run the evaluator against a small sample to confirm the output shape and the scoring distribution.

Typical use​

Autofill is most useful when:

  • You have a clear natural-language description of what to measure ("Is the response polite and accurate?").
  • The dataset and output schemas are stable and unlikely to change.
  • You want to bootstrap quickly and refine — not produce a final evaluator in one shot.

For one-off, very specific evaluators (e.g., regex matching a single field), it's often faster to write the evaluator inline or import a system template.