> ## Documentation Index
> Fetch the complete documentation index at: https://koreai.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# By Question Metric

The By Question metric evaluates how well agents respond to specific questions during customer interactions. Supervisors can apply this metric to all conversations (Static) or trigger it only when specific conditions occur (Dynamic). This metric helps teams measure response quality, enforce compliance, and provide targeted coaching.

By Question is available in two versions, **By Question V1** and **By Question V2**. You choose the version when you create the metric. V2 is an addition, not a replacement: existing V1 metrics continue to work unchanged, and you can still create new V1 metrics.

***

## Key Benefits

* Standardized quality assessment framework.
* Customizable evaluation criteria that fit your business needs.
* Multi-language support for global operations.
* Automated evaluation suggestions through AI.

## When to Use This Metric

| Use Case | Description |
| - | - |
| **Quality Assurance** | Evaluate whether agents follow required scripts, procedures, or expected responses. |
| **Training Assessment** | Measure how well agents apply training guidelines during customer conversations. |
| **Compliance Monitoring** | Measure how well agents deliver critical information such as disclaimers, privacy policies, or regulatory statements. |
| **Performance Standardization** | Apply consistent evaluation criteria across agents and interactions. |

***

## Choose a Version: V1 or V2

When you add a By Question metric, select the version that matches the complexity of the rule you need to evaluate.

* **By Question V1**: Separate trigger and response rules. Limited logic.
* **By Question V2**: Combined rule with condition and response. Clear justifications, supports Not Applicable and complex logic.

| Capability | By Question V2 | By Question V1 |
| - | - | - |
| **Rule Model** | A single unified rule describes the condition and the expected agent behavior together. | Configure Trigger and answer as two separate rules. |
| **Evaluation Flow** | The LLM evaluates in two steps — first whether the condition applies, then whether the agent complied. | Evaluate Trigger and answer independently, and then the application combines the results afterward. |
| **Not Applicable** | A first-class output with its own AI-generated justification. | Inferred by the application from the trigger result but no AI-generated explanation. |
| **Multi-condition Logic** | A single metric can express multi-condition logic. | Each condition needs its own individually configured metric. |
| **Justifications** | Separate, coherent justifications for the condition and the adherence verdict. | Two disconnected reasoning fragments. |
| **Evaluation Window** | Turn-based window: entire, first X, or last X turns. | Not supported. |
| **Conversation Context** | Optional. Reference a meta tag or custom field to evaluate behavior that depends on data outside the transcript. | Not available. |

![By Question V2](https://mintcdn.com/koreai/LNoBBV7dlD_8aUiO/ai-for-service/quality-ai/configure/evaluation-criteria/metrics-measurement-types/images/by-questions-v1-v2.png?fit=max\&auto=format\&n=LNoBBV7dlD_8aUiO\&q=85\&s=88ae234e2337e23197dabbbb26c736e9)

***

## How the Metric Works

The metric operates through a question-driven evaluation process with two main adherence approaches. Both approaches are available in V1 and V2.

* **Static Adherence**: Evaluates every response across all conversations. Use it for universal rules (for example, greetings). No triggers needed.

* **Dynamic Adherence**: Evaluates responses only when specific triggers or conditions occur. Use it for conditional rules that apply in certain situations.

***

## Configure a By Question Metric

These steps are common to both versions. The version-specific configuration follows in the next sections.

1. Navigate to **Quality AI** > **Configure** > **Evaluation Forms** > **Evaluation Metrics**.

2. Select **+ New Evaluation Metric**.

3. From the **Evaluation Metrics Measurement Type** dropdown, select **By Question**.

   ![Measurement Type](https://mintcdn.com/koreai/bPimY2iX8wmlOa2o/ai-for-service/quality-ai/configure/evaluation-criteria/metrics-measurement-types/images/by-question-dropdown.png?fit=max\&auto=format\&n=bPimY2iX8wmlOa2o\&q=85\&s=14252b69b95961bbd1af355253da357e)

4. Select the version — **By Question V1** or **By Question V2**.

5. Enter a descriptive **Name** (for example, `Agent's Warm Greeting`).

6. Select a **Language** used for evaluation.

7. Enter an evaluation **Question**, which is a reference prompt for supervisors during audits and reviews.

   ![Question and Adherence Type](https://mintcdn.com/koreai/bPimY2iX8wmlOa2o/ai-for-service/quality-ai/configure/evaluation-criteria/metrics-measurement-types/images/by-question-adhere-type.png?fit=max\&auto=format\&n=bPimY2iX8wmlOa2o\&q=85\&s=1075eaff444df24802294cc2ce7d87c0)

8. Select the **Adherence Type** (Static or Dynamic).

> [!NOTE]
>
> For **Static**, configure at least one agent answer utterance. For **Dynamic**, configure at least one trigger and one agent answer utterance.

***

## By Question V1 Configuration

V1 configures the trigger and the agent answer as two separate rules. Use V1 for straightforward checks that do not require Not Applicable justifications or multi-condition logic.

### Adherence Type Configuration

V1 supports two evaluation modes.

**Static Adherence**: Applies to all calls without any condition or trigger. Use for mandatory, universal compliance items like greeting scripts or regulatory disclaimers.

* Define acceptable utterances for the queue.
* Set a similarity threshold to evaluate whether the agent's response matches the predefined utterance.
* Configure at least one agent utterance template.

**Dynamic Adherence**: Evaluates adherence only after detecting a configured trigger. Use for context-sensitive checks.

* Define at least one trigger (customer or agent utterance) and one acceptable agent response.
* Set the similarity threshold by criticality:
  * **Lower threshold (\~60%, yellow)** — for casual interactions or greetings.
  * **Higher threshold (\~100%, green)** — for critical topics such as legal disclaimers or privacy policies.

### Trigger Configuration

Choose the utterance source that initiates the trigger:

| Trigger Type | Description |
| - | - |
| **Customer Utterance** | Triggered by something the customer says (for example, `I need a refund`) |
| **Agent Utterance** | Triggered by something the agent says (for example, `Can I transfer this call to support`) |

You can add multiple trigger utterances and define AND/OR conditions to control when the trigger activates.

### Trigger Detection Method

| Method | Description |
| - | - |
| **GenAI-Based** | Uses LLMs to detect intent contextually. No reference utterances or thresholds required. |
| **Deterministic** | Relies on predefined reference utterances and semantic similarity matching. |

### Agent Answer Configuration

#### GenAI-Based Adherence

Uses LLMs to detect meaning, context, and intent. Evaluates whether the agent's answer fulfills the intent, even if phrased differently.

1. Select **GenAI-Based Adherence** as the answer detection method.
2. Enter a **Description** explaining the metric's intent.

> [!NOTE]
>
> Before using GenAI-based adherence, confirm that supported models and the GenAI features (GenAI-based agent answer adherence and customer trigger detection) are enabled. No reference utterances or thresholds are required — LLMs evaluate adherence using zero-shot prompts.

#### Deterministic Adherence

Evaluates responses based on semantic similarity to predefined reference utterances.

1. Select **Deterministic Adherence**.
2. Define an **Answer** — acceptable utterances per queue. Use Generative AI to suggest similar variations.
3. Set the **Similarity** threshold:
   * Lower threshold (\~60%) for soft skills like greetings and etiquette.
   * Higher threshold (\~100%) for compliance-critical statements (Policy, Privacy, Disclaimers).

### Count Type Configuration

Choose how the metric evaluates adherence across the conversation:

**Entire Conversation**: Evaluates adherence throughout the entire interaction at any point.

**Time Bound**: Focuses on specific timeframes — the first or last X seconds or messages.

1. Select **Parameter** — First Part or Last Part of the conversation.

2. Set the evaluation window:
   * **Voice** — enter number of seconds.
   * **Chat** — enter number of messages.

3. Select **Create** to save and activate the metric.

***

## By Question V2 Configuration

V2 replaces the split trigger/answer architecture with a single unified rule definition. Instead of two disconnected rules, V2 describes the trigger condition and the expected agent behavior in one natural-language description. Use V2 when you need Not Applicable justifications or multi-condition logic.

### Unified Rule Definition

Define the condition and the expected agent behavior together in a single natural-language description rather than as separate trigger and answer rules. The auditor reads this description to return one of three verdicts for the conversation.

| Verdict | Meaning |
| - | - |
| YES | The agent satisfied the requirement. |
| NO | The agent violated the requirement. |
| NA | The requirement doesn't apply, or the conversation can't be evaluated reliably. |

### Conversation Context (Optional)

Conversation Context brings a value from outside the transcript into a V2 metric: a meta tag that a dialog sets during the conversation, or a custom field that the conversation carries from ingestion. Use it when the expected agent behavior depends on system data that the transcript doesn't show, such as an authentication status established in an IVR before a transfer.

The section sits between **Question** and **Description**, and it's off by default. It isn't available for By Question V1. With it off, the metric behaves exactly as it does today.

1. Turn on **Conversation Context (Optional)**.
2. For **Context field**, select **Meta tag** or **Custom field**.
3. Select the key from **Existing Meta tags** or **Existing Custom field**. The list displays keys that recent conversations carried.
4. Review the **Preview**, which displays the reference. For example, `{{meta.auth_status}}`.
5. Select **Insert into description** to place the reference in the **Description** at the cursor position.
6. Write what each value means, and the expected agent behavior for it, in the **Description**.

A metric can reference more than one context field, and one evaluation form can reference at most 10 distinct fields across all its metrics. Conversation Context applies to CCAI and Automation AI conversations.

> [!NOTE]
>
> The section displays a read-only **When the value isn't available** message. When Quality AI can't resolve a referenced key for a conversation, the metric returns Not Applicable automatically, with no configuration needed. See [Not Applicable Handling](#not-applicable-handling).

The **Description** field accepts up to 10,000 characters, which applies to every By Question V2 metric. The field uses an expandable editor with a character counter.

### Write the Metric Description (Best Practices)

Write the description as a set of labeled sections. The auditor reads these sections in a fixed order, so a clear, well-structured description produces consistent verdicts. Follow the structure, connectors, and style rules that follow.

#### Description Structure

A description uses up to four sections. Always write them in this order. Only `EXPECTED BEHAVIOR` is mandatory. `CONDITION`, `PROHIBITED`, and `EXCEPTION` are optional.

```text theme={null}
CONDITION: <when the metric applies>           # optional
EXPECTED BEHAVIOR: <what the agent must do>
PROHIBITED: <what the agent must never do>     # optional
EXCEPTION: <when evaluation can't be performed> # optional
```

| Section | Required | What it defines and how to write it |
| - | - | - |
| **CONDITION** | Optional | When the metric applies. Reference any participant and join items with AND or OR. Defaults to `Always.` when omitted. |
| **EXPECTED BEHAVIOR** | Mandatory | The observable actions the evaluated speaker must perform. Refer only to that speaker and join items with AND, OR, or THEN. |
| **PROHIBITED** | Optional | An action that fails the metric immediately, even when the expected behavior occurs. Listed items act as OR. |
| **EXCEPTION** | Optional | A situation that makes reliable evaluation impossible. The auditor checks it first and returns NA. Listed items act as OR. |

#### Simple and Extended Form

Use the simple form when a section holds a single statement (see Patterns A and B). Use the extended form when a section holds multiple components joined by connectors:

```text theme={null}
CONDITION:
  - Customer expresses frustration.
  AND
  - Customer mentions a billing issue.
EXPECTED BEHAVIOR:
  - Agent acknowledges the frustration.
  AND
  - Agent provides a next step.
EXCEPTION:
  - Customer disconnects before the agent responds.
```

#### Logical Connectors

Join multiple items with one of the following connectors. Use a single connector type per list.

| Connector | Meaning |
| - | - |
| **AND** | The agent must satisfy every listed item. |
| **OR** | The agent must satisfy at least one listed item. |
| **THEN** | The agent must perform the listed items in the stated order. |

> [!NOTE]
>
> Don't mix AND and OR in the same list. Mixed connectors make the logic ambiguous. If you need branching logic, create separate metrics instead.

#### How the Auditor Evaluates the Description

The auditor applies the sections in a fixed order and returns the first verdict that matches:

1. If any **EXCEPTION** matches, the verdict is **NA**.
2. If any **PROHIBITED** action occurs, the verdict is **NO**.
3. If the **CONDITION** is not met, the verdict is **NA**.
4. If the **EXPECTED BEHAVIOR** is fully satisfied, the verdict is **YES**.
5. Otherwise, the verdict is **NO**.

Because the auditor reads condition and behavior together, it produces one coherent explanation — and a clear reason when the metric doesn't apply.

#### Style Rules

* Write in the present tense and active voice.
* Describe observable actions, not subjective qualities.
* Refer to each speaker clearly so responsibilities are unambiguous.
* Write section headers and connectors in uppercase.
* End every statement with a period.
* Keep each bullet to **20 words or fewer**, and the full description to about **12 lines**.

#### Do and Don't

**Avoid subjective language.** Describe what the agent does, not how they seem.

```text theme={null}
Avoid:  EXPECTED BEHAVIOR: Agent is professional.
Use:    EXPECTED BEHAVIOR: Agent uses courteous language.
```

**Don't mix connectors in one list.** Mixed AND and OR make precedence ambiguous.

```text theme={null}
Avoid:  EXPECTED BEHAVIOR:
          - Agent offers refund.
          AND
          - Agent escalates.
          OR
          - Agent offers replacement.
```

**Don't use conditional branching.** Create separate metrics instead of "if/otherwise" logic.

```text theme={null}
Avoid:  EXPECTED BEHAVIOR:
          If customer agrees, agent refunds; otherwise escalates.
```

Also, avoid internal tool references, message identifiers, and turn identifiers in the description.

#### Standard Patterns

Start from one of these patterns and adapt it to your requirements.

**Pattern A — Trigger and response**

```text theme={null}
CONDITION: Customer requests assistance.
EXPECTED BEHAVIOR: Agent provides assistance.
EXCEPTION: Customer disconnects before response.
```

**Pattern B — Unconditional requirement**

```text theme={null}
CONDITION: Always.
EXPECTED BEHAVIOR: Agent greets the customer.
EXCEPTION: No agent messages exist.
```

**Pattern C — Sequential workflow**

```text theme={null}
CONDITION: Customer requests account information.
EXPECTED BEHAVIOR:
  - Agent verifies identity.
  THEN
  - Agent provides information.
EXCEPTION:
  - Customer refuses verification.
```

**Pattern D — Alternative acceptable responses**

```text theme={null}
CONDITION: Customer requests compensation.
EXPECTED BEHAVIOR:
  - Agent offers refund.
  OR
  - Agent offers replacement.
  OR
  - Agent escalates the issue.
EXCEPTION: Customer disconnects before response.
```

**Pattern E — Positive requirement with guardrails**

```text theme={null}
CONDITION: Always.
EXPECTED BEHAVIOR:
  - Agent communicates courteously.
PROHIBITED:
  - Agent guarantees approval.
  - Agent shares customer data.
EXCEPTION: None.
```

#### Description Template

Use this template as your starting point:

```text theme={null}
CONDITION: <Always | Trigger | AND/OR list>
EXPECTED BEHAVIOR:
  <Action or logical list>
PROHIBITED:
  - <Disallowed action>
EXCEPTION: <None | Situation | list>
```

### Not Applicable Handling

Not Applicable is a first-class output in V2. When an exception matches or the condition isn't met, the auditor returns Not Applicable directly, along with its own justification explaining why the metric did not apply. The application no longer infers Not Applicable from a trigger result.

Not Applicable also applies when a context field that the metric references can't be resolved for a conversation. In that case:

* Quality AI drops the affected rule before the model runs, and the model never receives an unresolved placeholder.
* The metric returns Not Applicable at the metric level, and the conversation's other metrics evaluate normally.
* The scoring follows the evaluation form's Not Applicable handling, either awarding points or removing the metric from scoring.
* An auditor can still score the metric manually, and the agent can't dispute the outcome.
* Quality AI records the reason in the conversation's Conversation Context log.
* The justification reads: "This check wasn't applicable because the required information (`<key>`) wasn't available for this conversation".

### Evaluation Window Constraints

V2 evaluates the rule against a turn-based window. Choose where in the conversation the metric applies:

* **Entire conversation** — evaluate at any point.
* **First X turns** — evaluate only the opening turns.
* **Last X turns** — evaluate only the closing turns.

A turn is one customer block and one agent block.

### Audit Screen (V2)

For V2 metrics, the Audit Screen displays two separate justification fields, giving evaluators a clear, auditable explanation for every outcome:

* **Condition determination** — why the condition did or didn't apply.
* **Adherence verdict** — whether the agent complied when the condition applied.

When the metric references a context field, a **Conversation Context** strip displays ahead of the justification. See [Conversation Context](https://koreai.mintlify.site/ai-for-service/quality-ai/analyze/ai-assisted-manual-audit#conversation-context-strip).

***

## Enable GenAI-Based Features (Prerequisites)

To activate GenAI-based features:

1. Navigate to **Manage** > **Generative AI** > **GenAI Features**.
2. Enable and publish the following features:

   * **GenAI-based agent answer adherence**
   * **GenAI-based customer trigger detection**

   ![GenAI-based Features](https://mintcdn.com/koreai/7ED-MzuzwUbgCHyq/ai-for-service/quality-ai/configure/evaluation-criteria/metrics-measurement-types/images/gen-ai-based-agent-answer-adherence-with-trigger.png?fit=max\&auto=format\&n=7ED-MzuzwUbgCHyq\&q=85\&s=4408d20e39b8fae930a1e2ecb8977b3b)

GenAI-based adherence uses Large Language Models (LLMs) to evaluate responses using intent and context. It doesn’t require example utterances or similarity thresholds, as adherence is evaluated using zero-shot prompts.

For guidance on writing effective prompts, refer to [AutoQA Prompting Guide](https://koreai.mintlify.site/ai-for-service/quality-ai/configure/evaluation-criteria/metrics-measurement-types/autoqa-prompting-guide).

***

## Edit or Delete a By Question Metric

1. Select the metric from the **By Question** category.

   ![Edit Metric](https://mintcdn.com/koreai/7ED-MzuzwUbgCHyq/ai-for-service/quality-ai/configure/evaluation-criteria/metrics-measurement-types/images/common-edit-eva-metics.png?fit=max\&auto=format\&n=7ED-MzuzwUbgCHyq\&q=85\&s=3e5270bde3ff0bfde2383636dbf3cec5)

2. Choose an option: **Edit** to modify the required details, or **Delete** to remove.

3. Select **Update** to save changes.

### Language Dependency Warnings

You can't remove a language while any evaluation form or metric uses it. Remove the language from all linked forms and metrics before deleting it.

![Language Warning](https://mintcdn.com/koreai/bPimY2iX8wmlOa2o/ai-for-service/quality-ai/configure/evaluation-criteria/metrics-measurement-types/images/by-question-lang-warning.png?fit=max\&auto=format\&n=bPimY2iX8wmlOa2o\&q=85\&s=619b92b850b6c8cf7e28fc7626bd80f2)

### Delete Warnings

Before deleting a metric:

1. Remove it from all associated evaluation forms.
2. If any attributes link to the metric, reassign them to a different metric first.
3. Resolve all dependencies before deleting the metric.

If the metric is still in use, the system displays a warning and prevents deletion.

***


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.