> ## Documentation Index
> Fetch the complete documentation index at: https://koreai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

Evaluations provide a structured framework to test, measure, and improve your agents by simulating conversations, scoring responses, and analyzing the results.

Create evaluation suites to test how your agent behaves across different scenarios, personas, and evaluators. Review the results to identify issues, validate changes, and continuously improve agent performance.

The platform supports both AI-assisted and manual evaluation workflows. We recommend that you use **Arch AI** to automatically generate evaluation suites based on your project. You can review and customize the generated suite before running it.

**Navigation**: Go to your project and select **Evaluate** > **Evals**.

## Evaluation workflow

**1. Create an evaluation suite**

Create an evaluation suite manually or let Arch generate one based on your project. Review and customize the suite before running it.

**2. Run evaluations**

Run the evaluation suite to simulate conversations and measure your agent’s performance across the configured scenarios, personas, and evaluators.

**3. Review Results**

Review evaluation scores, conversations, traces, and execution details to identify issues and opportunities for improvement.

**4. Repair and Optimize**

Use the evaluation results to identify recommended improvements. Depending on your Arch configuration, you can review changes manually or allow Arch to apply eligible improvements automatically.

**5. Validate Improvements**

Run the evaluation suite again to verify that the applied changes improve the agent's perfomance.

## Key concepts

| Concept | Description |
| :- | :- |
| **Evaluation Suite** | A collection of scenarios, personas, evaluators, and run settings that defines how an agent is evaluated. |
| **Eval Library** | Stores reusable personas, scenarios, and evaluators that can be shared across multiple evaluation suites within a project. |
| **Arch AI** | Generates evaluation suites, analyzes evaluation results, and recommends improvements as part of the evaluation workflow. |

## How Arch AI optimizes your agent

Arch AI helps you continuously improve your agents by analyzing evaluation results, generating repair recommendations, and validating improvements through repeated evaluation cycles.

```mermaid actions={false} theme={null}
%%{init: {'theme': 'base', 'themeVariables': {'background': 'transparent', 'primaryColor': '#EAF8E4', 'primaryTextColor': '#1a1a1a', 'primaryBorderColor': '#5CC83A', 'lineColor': '#5CC83A', 'secondaryColor': 'transparent', 'tertiaryColor': 'transparent', 'clusterBkg': 'transparent', 'clusterBorder': '#5CC83A', 'titleColor': '#1a1a1a', 'clusterLabelBackground': 'transparent', 'edgeLabelBackground': '#EAF8E4'}}}%%
flowchart LR
    A[Evaluate]
    B[Analyze]
    C[Recommend]
    D[Apply]
    E[Validate]

    A --> B --> C --> D --> E
```

| Stage | What Happens |
| :- | :- |
| **Evaluate** | Run the evaluation suite to measure your agent's performance across the configured scenarios and personas. |
| **Analyze** | Arch AI analyzes evaluation scores, conversation traces, and execution details to identify failed or low-scoring conversations. |
| **Recommend** | Arch AI generates recommendations to improve prompts, workflows, tool usage, or agent behavior based on the identified issues. |
| **Apply** | Review and apply recommended changes based on your configured Arch autonomy level. |
| **Validate** | Re-run the evaluation suite to verify that the applied changes improve the evaluation results. |

**Learn more**: To see where Evaluations fit in the Arch AI lifecycle, see [Arch AI](https://koreai.mintlify.site/agent-platform/arch-ai.md#lifecycle-phases).

### Example: Improve tool call accuracy using Arch

Suppose an evaluation shows that your agent has a low Tool Call Accuracy score. Arch AI analyzes the evaluation results and conversation details to understand why the agent is failing. It may identify that the agent is selecting the correct tool but providing incorrect parameters because the instructions do not clearly specify the required inputs.

Based on this analysis, Arch AI recommends an improvement to the agent’s instructions. After the change is reviewed and applied, the evaluation suite is run again to validate the improvement. If the agent now provides the correct parameters and the Tool Call Accuracy score improves, the evaluation cycle is complete.

**Learn more:** For more information about the Arch AI reinforcement loop and continuous optimization, see [Optimize with Arch AI](https://koreai.mintlify.site/agent-platform/optimize.md).

***

## Create evaluation suites

Evaluation suites define how your project is evaluated. Each suite combines scenarios, personas, evaluators, and run settings to measure your agent's performance.

You can create an evaluation suite in one of the following ways:

| Option | Description |
| :- | :- |
| [Create with Arch](#create-eval-suite-with-arch) (Recommended) | Let Arch generate an evaluation suite based on your project. |
| [Create Agentic Testing Suite](#create-eval-suite-manually) | Create and configure an agentic evaluation suite manually using scenarios, personas, and evaluators. |
| [Create RAG Testing Suite](#create-a-rag-evaluation-suite) | Create a RAG evaluation suite using a CSV dataset to evaluate an agent's retrieval and response quality. |

### Create eval suite with Arch

Use **Create with Arch** to automatically generate an evaluation suite based on your project.

1. Go to **Evaluate** > **Evals**.
2. Select **Create with Arch**.
3. Wait for Arch to generate the evaluation suite.
4. Review the generated evaluation suite.
5. (Optional) Select **Edit Configuration** to customize the generated components.
6. Select **Create & Run Suite** to create the suite and start the evaluation.

> [!NOTE]
>
> Arch analyzes your project and automatically generates the evaluation suite components, including scenarios, personas, evaluators, and run settings. You can review and modify the generated configuration before running the evaluation.

### Create a RAG evaluation suite

Create a RAG evaluation suite to evaluate an agent's retrieval and response quality using a CSV dataset of test questions. Unlike conversation evaluation suites, RAG suites evaluate individual questions against a single agent without using personas or scenarios.

For detailed instructions, see [RAG evaluation suites](#rag-evaluation-suites).

### Create Agentic testing suite

To create an evaluation suite manually, from the **Evals** page, select **Create Agentic testing suite** or **+ New suite**. Both options open the **Create eval suite** configuration screen.

Configure the suite by defining its:

* **Basics** — Specify the suite name, description, scope, and modality.
* **Arch autonomy** — Select how much autonomy Arch has when applying improvements and set the maximum number of runs per loop.
* **Scenarios** — Select or add the scenarios to evaluate.
* **Personas** — Select or add the personas that simulate users.
* **Evaluators** — Select evaluators and configure their weights and passing score.
* **Run settings** — Configure how conversations are generated and when the suite runs.

Select **Create & run suite** to create the evaluation suite and start the evaluation.

***

## Evaluation suite components

Every evaluation suite, whether created manually or with Arch, contains the following components. Together, they define what is evaluated, how the evaluation is executed, and how the results are scored.

When you create an evaluation suite with **Arch**, these components are generated automatically based on your project. You can review and modify them before running the evaluation.

| Section | Description |
| :- | :- |
| **Scenarios** | Defines the user tasks or conversations used to evaluate your agent. |
| **Personas** | Defines the characteristics and behavior of users participating in the evaluation. |
| **Evaluators** | Defines the metrics and criteria used to score each conversation. |
| **Run Settings** | Configures how the evaluation is executed, including conversation variations and execution options. |

### Basics

Use the Basics section to define what the evaluation suite covers and which project state it evaluates.

* **Suite name** — Enter a name for the evaluation suite.
* **What this suite covers** — Describe the purpose or focus of the evaluation suite.
* **Test scope** — Choose whether to evaluate the entire project or specific agents.
* **Version to test** — Select the project version, deployment, or environment to evaluate. Depending on the selected scope, the available options may include the Working copy, an Environment, a specific Deployment, or a published Agent version.

Select **Change scope** to change the project or agent scope and configure the corresponding evaluation target.

### Arch's autonomy level

Arch's autonomy level setting determines how much control Arch has to apply improvements and re-run evaluations when a run does not meet the passing score. You can change this setting at any time.

* **Autopilot** — Arch fixes and re-runs the evaluation on its own. Arch patches the agent and keeps re-running until the score clears the target or the configured loop limit is reached.
* **Copilot** — Arch automatically applies low-risk fixes. Arch diagnoses regressions, applies low-risk changes, and stages higher-risk changes for approval.
* **Advisor** — Arch recommends fixes only. Arch explains what regressed and proposes changes without applying them.

**Max runs per loop**: Sets the maximum number of fix-and-rerun attempts Arch can make, even if the score has not cleared the target. This limits the number of attempts Arch can make in a single loop.

***

## Scenarios

Scenarios define the conversation flow, user intent, and expected outcomes used during evaluations.

Each scenario represents a conversation flow used to evaluate how the agent handles specific tasks, behaviors, or outcomes.

To create a scenario:

1. In the **Scenarios** section, select **Add Scenario**.
2. Complete the scenario details.
3. Select **Create**.

| Field | Description |
| :- | :- |
| **Name** | Unique name for the scenario. |
| **Description** | Brief description of what the scenario tests. |
| **Category** | Logical grouping used to organize scenarios, such as Billing, Customer Support, Technical, or Onboarding. |
| **Difficulty** | Complexity level of the scenario: **Easy**, **Medium**, or **Hard**. |
| **Entry Agent** | Agent that starts the conversation. Primarily used in multi-agent projects. |
| **Max Turns** | Maximum number of conversation turns before the evaluation stops. |
| **Initial Message** | First user message that starts the conversation. |
| **Session Variables (JSON)** | Optional session variables used to provide contextual data for the scenario. Scenario session variables are merged with the persona's session variables when an evaluation runs. If the same key is defined in both, the scenario value takes precedence unless the key is selected as a **Priority key** for the persona. |
| **Instructions** | Optional guidance sent to the simulated-user LLM about how the user should behave. |
| **Scripted values JSON** | Optional key-value pairs for exact values the simulated user should provide when relevant, such as a PIN or account number. The LLM does not see the literal values. Supported for text scenarios only. See [Personas](https://koreai.mintlify.site/agent-platform/evals.md#personas) to define reusable scripted values for a persona. |
| **Agent Path** | Expected sequence of agent handoffs during the conversation. Applicable to multi-agent projects. |
| **Expected Milestones** | Key checkpoints the conversation should achieve. |
| **Expected Outcome** | Describes what a successful conversation should accomplish. |
| **Tags** | Labels used to organize and filter scenarios by feature area, regression suite, priority, or other categories. |

> [!TIP]
>
> Judges score what can't be checked exactly. Use **Agent Path** and **Expected Milestones** for everything that must be exact.

**Scripted values** let you define exact facts for the simulated user to provide when relevant, such as a PIN or account number. They are separate from **Session Variables**, which provide context to the agent. Scripted values are supported for text scenarios only.

For example, enter the following in **Scripted values JSON**:

```yaml theme={null}
{
  "pin": "4417",
  "account_number": "00123456789"
}
```

![Evaluate](https://mintcdn.com/koreai/x0g9paEmDJxRcPCe/agent-platform/images/scenario.png?fit=max\&auto=format\&n=x0g9paEmDJxRcPCe\&q=85\&s=341ebd47b042ecbf9a75385d77afc3ce)

### Example scenario

```yaml theme={null}
SCENARIO:
  name: "Flight Rebooking After Cancellation"
  category: booking
  difficulty: medium

  initial_message: >
    My flight was cancelled and I need to rebook for tomorrow.

  expected_outcome: >
    Agent identifies the cancelled booking, offers alternatives,
    and confirms a new flight.

  max_turns: 15

  expected_milestones:
    - "Identify cancelled flight"
    - "Present rebooking options"
    - "Confirm new booking"

  agent_path:
    - "Supervisor"
    - "Booking_Manager"
```

### Troubleshoot scenarios

| Issue | Recommendation |
| - | - |
| Duplicate name error | Persona and scenario names must be unique within a project. Use a more specific name or delete the existing one. |
| Persona not behaving as expected in evals | Refine the **Goals** and **Constraints** fields. These are used as system prompt instructions for the simulated user LLM. |
| Scenario timing out | Increase the **Max Turns** value or simplify the expected conversation path. |

***

## Personas

Personas represent different types of users who interact with your agent.

Each persona simulates unique communication styles, domain expertise, goals, behaviors, and constraints to help test how the agent performs across varied user interactions.

To create a persona:

1. In the **Personas** section, select **Add Persona**.

2. Complete the persona details.

   | Field | Description |
   | - | - |
   | **Name** | Unique name for the persona. |
   | **Description** | Brief description of the persona and the type of user it represents. |
   | **Communication Style** | Defines how the persona communicates, for example, casual, terse, or formal. |
   | **Domain Knowledge** | Defines the persona's familiarity with the subject, such as beginner, intermediate, or expert. |
   | **Behavior Traits** | Characteristics that influence how the persona behaves during the conversation. For example, asks follow-up questions, polite, or impatient. |
   | **Goals** | Defines what the persona is trying to accomplish during the interaction. |
   | **Constraints** | Rules or limitations that influence the persona's behavior during the conversation. |
   | **Adversarial Type** | Specifies whether the persona behaves as a normal user or intentionally challenges the agent to test its robustness. |
   | **Session Variables (JSON)** | Optional session variables that provide contextual data for the persona. |
   | **Priority keys** | Select session variable keys whose persona values take precedence over values defined in a scenario. |
   | **Instructions** | Optional rules for the simulated user to follow, such as the order in which to provide scripted values. |
   | **Scripted values JSON** | Optional JSON values the simulated user provides exactly as written when relevant. The persona model does not see the values. For partial values, such as a card’s last four digits, add them as a separate key. By default, the scenario’s value takes precedence when the same key is defined in both places. See [Scenarios](https://koreai.mintlify.site/agent-platform/evals.md#scenarios) to define scenario-specific scripted values. |
   | **Priority scripted keys** | Select the keys for which the persona’s value should override the scenario’s value. |

3. Select an adversarial behavior type if you want to simulate edge cases or malicious interactions.

4. Click **Create**.

![Evaluate](https://mintcdn.com/koreai/mOz_ZNDanBGKqV7f/agent-platform/images/evaluate.png?fit=max\&auto=format\&n=mOz_ZNDanBGKqV7f\&q=85\&s=a4fba7023ccbc04b33cfd0cd43e53a5c)

### Example persona

```yaml theme={null}
PERSONA:
  name: "Impatient Business Traveler"
  communication_style: terse
  domain_knowledge: expert
  behavior_traits:
    - impatient
    - asks_follow_up_questions
  goal: "Rebook a cancelled flight quickly"
  constraint: "Avoid unnecessary conversation"
```

### Adversarial persona types

You can simulate adversarial or edge-case user behaviors using the **Adversarial Type** field.

To test agent safety and robustness:

1. Enable **Adversarial** while creating a persona.
2. Select the adversarial type.

| Type | Purpose |
| - | - |
| Prompt Injection | Attempts to override agent instructions |
| Social Engineering | Attempts to extract sensitive information |
| Off-topic Derailer | Redirects conversations away from the intended agent goal |
| Abusive User | Uses hostile or inappropriate language |
| Edge Case Explorer | Sends unusual or unexpected inputs (empty, very long messages) |

### Troubleshoot personas

| Issue | Recommendation |
| - | - |
| The persona's behavior is inconsistent | Refine the goals and constraints fields |
| The persona's responses are unrealistic | Add more specific behavioral traits and communication styles |
| The persona is too passive or aggressive | Adjust goals, constraints, and adversarial settings |

***

## Evaluators

Evaluators define how conversations are assessed during an evaluation. Each evaluator measures a specific aspect of the conversation, such as response quality, safety, task completion, or compliance, and assigns a score based on the configured evaluation criteria. To configure an evaluator, follow these steps:

1. In the **Evaluators** section, select **Add Evaluator**.
2. Enter the evaluator name and description.
3. Select the **Type** and **Category**.
4. Configure the evaluator based on the selected type.
5. Select **Create**.

> [!NOTE]
>
> Lower evaluator temperatures typically produce more consistent scoring results.

### Evaluator types

Supported evaluator types include:

| Type | Description |
| :- | :- |
| LLM Judge | Uses an LLM to evaluate conversations. |
| Code Check | Uses deterministic programmatic scoring. |
| Trajectory | Evaluates conversation flow and milestones. |
| Human Review | Flags conversations for manual review. |

### LLM Judge evaluators

An LLM Judge evaluator uses a separate LLM to assess the quality of agent responses based on a scoring rubric you define.

| Field | Description |
| :- | :- |
| **Judge Model** | Language model used to evaluate the conversation. |
| **Temperature** | Controls the randomness of the judge model's responses. Lower values produce more consistent evaluations. |
| **Judge Prompt** | Instructions that define the evaluation criteria for the judge model. |
| **Chain-of-Thought Reasoning** | Enables the judge model to perform intermediate reasoning before assigning a score. |
| **Scale Type** | Specifies the scoring method: **Pass/Fail** or **1–5 Scale**. |
| **Bias Mitigation** | Applies techniques to help reduce bias and improve evaluation consistency. |

![Evaluate](https://mintcdn.com/koreai/x0g9paEmDJxRcPCe/agent-platform/images/llm-judge.png?fit=max\&auto=format\&n=x0g9paEmDJxRcPCe\&q=85\&s=95663a2d83c2ea32a146397732345b98)

#### Write effective judge prompts

The judge prompt is one of the most important evaluator configurations. Well-defined prompts produce more consistent and reliable evaluation results. Effective judge prompts:

* Clearly define evaluation criteria
* Focus on observable behavior
* Avoid ambiguous language
* Include examples when possible

#### Example judge prompt

```yaml theme={null}
JUDGE_PROMPT:
  You are evaluating an AI agent's response quality in a customer support context.

  Evaluate the conversation on these criteria:
    1. Did the agent correctly identify the customer's intent?
    2. Did the agent provide accurate information?
    3. Did the agent follow the expected conversation flow?
    4. Was the agent's tone appropriate and professional?

  Score each conversation using the provided rubric.

  Focus on the agent's responses, not the simulated user's messages.
```

#### Configure bias mitigation

LLM judges can exhibit scoring biases. Use bias mitigation settings to improve evaluation consistency and reliability.

| Setting | Description | Default |
| :- | :- | :- |
| Position Swap | Evaluates the conversation in both original and reversed order to reduce positional bias. | On |
| Blind Evaluation | Removes agent or persona identity information before judging. | On |
| Cross-Model Judge | Uses a different model family than the agent being evaluated. | Off |
| Evidence-First (RULERS) | Requires the judge to cite evidence before assigning scores. | On |

### Trajectory evaluators

Trajectory evaluators assess the agent's execution behavior rather than response quality.

Use them to validate:

* Milestone completion -- did the conversation hit expected checkpoints?
* Handoff correctness -- did the supervisor route to the right agent?
* Path efficiency -- how many unnecessary steps did the agent take?
* Tool sequence -- did the agent call tools in the right order?

### Code Check evaluators

Use Code Check evaluators for deterministic validations that do not require an LLM. They use built-in scorers to validate agent responses and runtime behavior.

Typical use cases include:

* Regex matching
* Keyword validation
* Latency or response-time thresholds
* Structured output validation

For text-agent latency, select one of these built-in scorers and set its **Max ms** budget:

| Built-in scorer | What it measures |
| - | - |
| **First response time** | Runtime processing time for the first agent reply. |
| **Average response time** | Average runtime processing time across agent replies in the conversation. |
| **First response time to first token** | Time until the first streamed output for the first agent reply. |
| **Average time to first token** | Average time until the first streamed output across agent replies in the conversation. |

A latency check passes when the measured time is within the **Max ms** budget and fails when it exceeds the budget. It returns a score of 1 when it passes and 0 when it fails. These measurements cover Runtime processing time; they do not include network time, evaluation queue time, or screen rendering.

### Human Review evaluators

Use Human Review evaluators for subjective or manual quality assessments.

Human Review evaluators flag conversations for manual inspection when evaluation scores fall below configured thresholds, allowing reviewers to validate agent behavior, response quality, or policy compliance before approval or release.

> [!NOTE]
>
> A Human Review evaluator leaves a run provisional. Keep that suite out of automated baselines.

### Scoring scale types

The scoring rubric defines how the evaluator assigns scores to conversations. It supports Likert and Binary scales.

#### Likert scale

Use a 1 to 5 scale to define detailed evaluation criteria for each score level.

| Score | Description |
| - | - |
| 5 - Excellent | Addresses the user's request with accurate and complete information. |
| 4 - Good | Addresses the request with minor omissions. |
| 3 - Adequate | Partially addresses the request but misses important details. |
| 2 - Poor | Mostly misses the request or provides inaccurate information. |
| 1 - Failing | Fails to address the request or provides harmful information. |

#### Binary scale

Use pass or fail scoring for binary evaluation criteria.

| Score | Description |
| - | - |
| 1 - Pass | The agent completes the task within the expected flow. |
| 0 - Fail | The agent fails to complete the task or deviates from expected behavior. |

### Troubleshoot evaluators

| Issue | Recommendation |
| :- | :- |
| Inconsistent scores across runs | Lower evaluator temperature (try `0.1`) and enable evidence-first mode. Run multiple variants per evaluation to get statistical confidence. |
| Judge ignores rubric criteria | Make rubric instructions more specific using examples. |
| The Judge model is too expensive | Use a smaller model for initial screening and reserve larger models for detailed analysis. Set appropriate `maxTokens` limits. |
| The Evaluation cost is too high | Use smaller judge models during development. |
| Scores appear random | Increase statistical sample size using variants. |

***

## Configure run settings

Run settings determine how the evaluation suite is executed. To configure, follow these steps:

1. In the **Run Settings** section, configure the run settings.
2. Select **Create & Run Suite**.

### Run settings

| Setting | Description |
| - | - |
| **Variations per pair** | Specifies the number of conversation variations to generate for each scenario and persona combination. Multiple variations help evaluate the consistency of your agent's responses across different interactions. Default is 3. Use one variation for suites that write real rows, and reset the rows before each run. |
| **Conversation model** | Specifies the model used to generate the conversations that are graded. |
| **Conversation budget** | Sets the spending limit for the conversation model. You can set the budget in dollars or tokens, or leave it unlimited. |
| **When to run** | Specifies whether the evaluation suite runs manually or on a recurring schedule. For scheduled runs, configure the frequency (Daily, Weekly, or Monthly), day and time, start date, and end condition. The next scheduled run is shown in the eval suite, and scheduled runs are identified in the run history. |
| **Turn time limit** | Set the maximum time allowed for a turn, in milliseconds. A turn that exceeds the limit fails the case for latency. Scenarios with their own turn limit keep that limit. If no limit is set, no turn time limit is applied. |
| **Max parallel calls** | Specifies the maximum number of conversations that can run concurrently. |
| **Retries on failure** | Specifies the number of times to retry a conversation when it fails. |

> [!NOTE]
>
> The total number of conversations in an evaluation is calculated as: **Scenarios × Personas × Variations**

### How evaluations are executed

During execution:

* Every selected Persona interacts with every selected Scenario
* Each conversation is independently executed
* All configured Evaluators score the resulting conversations

This creates a full evaluation matrix across personas, scenarios, and evaluators.

In text evaluations, each repeated message from the simulated user reaches the agent as a separate turn, even when the message is identical to the previous one.

**Example Evaluation Matrix**

```yaml theme={null}
EVAL_SET:
  personas: 3
  scenarios: 4
  evaluators: 2
  variants: 2

TOTAL_CONVERSATIONS:
  formula: "3 Personas × 4 Scenarios × 2 Variants"
  result: 24

TOTAL_EVALUATIONS:
  formula: "24 Conversations × 2 Evaluators"
  result: 48
```

Each conversation is executed as an independent multi-turn session where the persona LLM simulates the user according to the scenario definition.

***

## View evaluation

After you create and run an evaluation suite, the Evals page displays all evaluation suites in your project. Use this page to monitor suite performance, review evaluation coverage, and open a suite to view detailed results.

Each evaluation suite displays:

* **Behavior Score** – Overall evaluation score for the latest run.
* **Coverage** – Number of scenarios, personas, variations, and generated conversations included in the evaluation.
* **Evaluators** – Number of evaluators configured for the suite.
* **Last Run** – Shows when the suite was last run and whether it runs manually or on a schedule.

Select an evaluation suite to view its detailed results. The evaluation suite page includes the following tabs: Overview, Latest run, and History.

### Overview

Provides a high-level summary of the evaluation suite and its latest execution. Use it to monitor overall performance, review evaluation coverage, and identify areas that require attention.

| Overview item | Description |
| - | - |
| **Run status and progress** | Shows the status of the latest run and its progress. |
| **Behavior score and trend** | Shows the latest behavior score and how the score changes across runs. |
| **Latest run summary** | Provides details about the most recent evaluation run. |
| **Arch recommendation** | Shows the status of recommendations from Arch based on the evaluation results. |
| **Evaluators below target** | Highlights evaluators whose results are below the configured passing score. |

### Alerts

Use the Alerts section on a suite’s Overview page to get notified when a watched run fails, falls below its target score, or regresses.

Alerts apply to the suite as a whole; they do not identify a particular scenario, persona, or evaluator as the cause.

To set up alerts for an evaluation suite, select *Create alert* to add a rule for that suite and configure the rule:

| Field | Description |
| - | - |
| **Rule name** | Name the alert so you can identify it in the suite and alert inbox. |
| **Evaluating** | Turn the rule on or off. Turning it off pauses evaluation but keeps the rule in the list. |
| **Alert when** | Choose **Run failed** if a run does not complete successfully, **Score below target** if its score falls below the suite target, or **Regressed against previous run** if its score drops beyond the suite’s regression tolerance. |
| **Runs to watch** | Choose **Scheduled**, **CI/CD**, **Manual**, or any combination. Only runs started from the selected sources can trigger the alert. |
| **Cooldown** | Set how long the rule stays quiet after firing before a later run can trigger it again. |
| **Deliver via** | Choose **Email** or **Webhook**. Email requires recipient addresses. Webhook requires a URL and can use an auth profile. |

For regression alerts, the regression note shows the suite’s tolerance and links to the suite settings. When a watched run matches the rule, the alert appears in the project’s alert inbox with a reason, and the platform sends a notification through the configured delivery method.

Evaluation alerts support failed runs, below-target scores, and regressions. They do not currently alert on run cost or duration.

***

### Latest run

Displays the results of the most recent execution of the evaluation suite. Use this page to:

* Review the overall run score and evaluator results.
* Review the evaluator breakdown.
* View individual evaluation conversations and their results.
* Search and filter conversations.
* Retry conversations that encountered errors using Retry all errors.

***

### History

The History tab displays previous evaluation runs for the suite. Use it to review how the suite's results change across runs and understand the changes associated with each run.

Includes information such as:

* Run date and duration
* Changes associated with the run
* Behavior score and source metric
* Source of the run
* View conversations to review conversations from a specific run

***

## Analyze evaluation results

After an evaluation completes, review the results to understand how your agent performed and identify opportunities for improvement.

You can:

* Review conversation transcripts.
* Analyze evaluator scores and reasoning.
* Inspect execution traces and tool usage.
* Compare expected and actual conversation outcomes.
* Identify patterns across successful and failed conversations.

### Analyze conversations

Select a conversation from the **Latest Run** or **History** tab to review the evaluation details.

For each conversation, you can inspect:

| Information | Description |
| - | - |
| **Transcript** | Complete conversation between the persona and the agent. |
| **Evaluator Scores** | Scores assigned by each evaluator, including reasoning when available. |
| **Execution Trace** | Tool calls, handoffs, workflow execution, and other runtime details. |
| **Milestones** | Expected milestones that were completed or missed. |
| **Tool Usage** | Tool invocations, inputs, outputs, and any execution failures. |

### View evaluator reasoning

When **Chain-of-Thought Reasoning** is enabled for an LLM Judge evaluator, the evaluation results include reasoning that explains how the score was determined.

Use evaluator reasoning to identify improvements in:

* Agent instructions
* Workflows
* Tool configuration

An example evaluator reasoning:

```yaml theme={null}
evaluation_result:
  score: 3/5

  reasoning: >
    The agent correctly identified the customer's intent
    but failed to provide complete rebooking options
    before requesting additional information.
```

### Act on results

Use the evaluation results to prioritize improvements to your agent.

| Finding | Recommended Action |
| :- | :- |
| **Low quality scores** | Refine agent instructions, goals, personas, or workflow logic. |
| **Low safety scores** | Strengthen guardrails and evaluate with adversarial personas. |
| **Low tool accuracy** | Review tool configuration, input parameters, and agent instructions. |
| **Handoff issues** | Review handoff conditions and validate the expected agent path. |

### Compare runs and set a baseline

Studio has no Compare screen. The Arch MCP `platform_eval_runs` compare operation returns evaluator-level averages for two complete, compatible runs. Open the run cases to see scenario-level results.

A baseline is a complete run that records the starting point of a stable suite. Set it as the suite's `baselineRunId` through `platform_eval_sets` or the API. Comparison is refused when results are incomplete or the scenarios, evaluators, or variations changed. After you change the suite on purpose, create a new baseline.

The platform doesn't measure how often an LLM Judge agrees with people. Before you use a judge as a release gate, hand-score 10 to 20 conversations and compare.

> [!TIP]
>
> For baselines, judge calibration, and CI gates, see [Evaluate agents](https://koreai.mintlify.site/agent-platform/best-practices/evaluate-agents.md).

### Troubleshoot

| Issue | Recommendation |
| :- | :- |
| **Scores appear inconsistent** | Increase the number of **Variations per Scenario × Persona** and lower the judge model temperature for more consistent results. |
| **Unexpectedly high scores** | Review the evaluator prompt and scoring criteria to ensure they accurately reflect the expected behavior. |
| **Higher execution costs** | Reduce the number of variations or use smaller judge models during development. |

***

## RAG evaluation suites

A RAG evaluation suite evaluates an agent's ability to retrieve relevant information and generate accurate answers. Unlike conversation evaluation suites, which use personas and scenarios to simulate conversations, RAG evaluation suites use a CSV dataset of test questions to evaluate a single agent's retrieval and response quality.

### Create a RAG evaluation suite

1. Go to **Evaluate > Evals** and select the option to create a RAG evaluation suite.
2. Under **Basics**, enter a suite name and, optionally, a description. Select the target agent and version.
3. Under **Evaluation data**, upload a CSV file containing your test questions.
4. Under **Evaluators, weights & passing score**, add the evaluators you want to use. Configure their weights and passing scores, and set the suite target score.
5. Configure the run settings and select **Create & run suite**.

A RAG suite evaluates individual questions against the selected agent. It does not use personas, scenarios, or conversation simulations.

### Prepare the evaluation dataset

Prepare a CSV file with the following columns. Column names must match the sample field names.

| Column | Description |
| - | - |
| `user_input` | The question sent to the agent. Required. |
| `response` | The expected or supplied answer. Optional. If empty, the agent's answer is generated during the run. |
| `ground_truth` | The correct answer. Optional. It cannot be generated; metrics that require it are skipped for samples without it. |
| `actual_contexts` | Retrieved context passages in retrieval-rank order. Optional. |
| `actual_context_ids` | IDs of retrieved context chunks in retrieval-rank order. Optional. |
| `expected_context_ids` | IDs of the context chunks that should be retrieved. Optional. |

The agent can generate the answer and retrieve context when these fields are not provided. Supplied answers and contexts are used during evaluation instead of being generated.

**CSV format requirements:**

* Enter list fields (`actual_contexts`, `actual_context_ids`, and `expected_context_ids`) as JSON arrays of strings. For example, `["chunk 1","chunk 2"]`.
* The first element represents retrieval rank 1.
* Values can contain commas, quotes, and line breaks.
* Leave an optional field empty if it is not provided for that row.

The uploaded CSV is checked against the specified format before it is used. The `user_input` column is required.

The platform uses the uploaded CSV as provided. It does not maintain a dataset library or version uploaded datasets. To use different evaluation data, upload a new CSV.

**Dataset limits:** A CSV can contain up to 10,000 rows and must not exceed 20 MB.

### Configure evaluators and scoring

RAG evaluation supports evaluators from the **LLM Judge** and **Code Check** families.

* **LLM Judge:** Context Precision, Context Recall, Faithfulness, Factual Correctness, and Noise Sensitivity.
* **Code Check:** Precision\@K, Recall\@K, and Mean Reciprocal Rank (MRR).

Evaluator availability depends on the fields provided in the CSV and the fields the agent can generate. Evaluators that cannot be used with the available data are unavailable, with a reason.

Configure a weight and passing score for each evaluator. The evaluator weights must total 100. You can distribute the weights equally.

The **suite target score** defines the overall score the suite is expected to achieve. It is different from the passing score configured for each evaluator.

**Evaluator limit:** You can add up to 12 evaluators to a suite.

### Run a RAG evaluation suite

A RAG evaluation run processes each CSV row as an individual sample. For each sample, the run uses the supplied answer and retrieved contexts when available. Otherwise, it runs the target agent to generate them.

If a metric requires data that is unavailable for a sample, that metric is skipped for that sample, with a reason. The rest of the evaluation continues.

You can run a suite manually or configure a recurring schedule. Run settings also include advanced execution options.

### Understand evaluation scores

Each evaluator produces a score for each sample. The evaluator's passing score determines whether a sample passes or fails that evaluator.

Evaluator weights determine how much each evaluator contributes to the overall sample score. If an evaluator is skipped for a sample, the remaining weights are adjusted for that sample.

The run's overall score is the average of the weighted sample scores. The suite target score determines whether the run meets the suite's target.

Most RAG metrics use a higher-is-better scoring direction. Noise Sensitivity uses a lower-is-better direction. Scores are normalized before aggregation so that higher values always represent better performance.

***

**Related articles:**

* [Arch AI](https://koreai.mintlify.site/agent-platform/arch-ai.md)
* [Create agents](https://koreai.mintlify.site/agent-platform/create-agents.md)
* [Optimize with Arch AI](https://koreai.mintlify.site/agent-platform/optimize.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.