Scheduled evaluations
For applications already in production, you don't want to wait for someone to click "Run evaluation" — you want continuous quality signal. Scheduled evaluations run automatically on a cron, over a sampled portion of recent live runs.
When to run the evaluation
When an evaluation is configured to apply to future runs, you choose when those runs are evaluated:
- Run immediately — as soon as a completed run arrives, trigger the evaluation. No batching.
- Run on a schedule — the system runs the evaluation job at configured intervals and evaluates all pending completed runs that have not been evaluated since the last scheduled run.
"Run on a schedule" is the right choice for cost-controlled continuous evaluation — process a batch of runs at a daily or weekly cadence rather than evaluating every single run as it arrives.
Schedule configuration
You can configure the schedule in two ways.
Simple schedule (time-based)
Pick a frequency and provide the required parameters.
| Frequency | Fields | Example |
|---|---|---|
| Hourly | Every __ hour(s) | Every 3 hours |
| Daily | Every __ day(s) + At HH:MM | Every 2 days at 04:20 |
| Weekly | Every __ week(s) + On <day> + At HH:MM | Every 1 week on Monday at 04:20 |
Advanced schedule (CRON)
For arbitrary schedules, use a full cron expression.
- Cron expression — standard 6-field cron (seconds, minutes, hours, day-of-month, month, day-of-week). A short field-guide legend is shown in the UI.
- Template picker — "Choose a template to get started" with common cron patterns.
- Timezone — required dropdown.
- Preview — a read-only section shows the next few upcoming run times (next 5 schedules) based on your cron + timezone, so you can verify you got the expression right.
Timezone
- The schedule runs in the timezone you select (default: workspace/user timezone if available).
- The chosen timezone applies to all Daily/Weekly times and to interpreting the "next run" preview.
Sampling
Evaluating every live run is expensive and often unnecessary. When configuring a scheduled evaluation against future traces, set a sampling percentage.
- Range: 1–100 (integers only).
- The percentage applies to the runs filtered by the status filter you've selected (e.g., completed runs only).
- Only the sampled percentage is evaluated; the rest of the runs aren't sent through the evaluators.
This is how you get representative quality signal without paying for full-traffic evaluation.
Each scheduled run
Each fire of a scheduled evaluation creates a separate test run in the test runs table, with timestamp and version metadata. Results land in the standard Excel-like results view — see Results table.
If a scheduled run fails (e.g., no eligible runs in the window, evaluator service error), the failure is logged and visible in the UI; the next scheduled fire still runs on its own cadence.
Pairing with multi-version comparison
Scheduled evaluations cover one application version per schedule. To compare across versions, set up a separate schedule per version and use Multi-version comparison to view them together.