Measure and Improve Your Agent

Useful measurement begins with a business question. Decide what outcome matters, which interactions belong in the population, and what evidence proves success before building a dashboard or starting an experiment.

​

Start with the observation you need

• Use a conversation when each completed interaction is one observation.
• Use a Task attempt when unanswered or failed attempts belong in the population.
• Use one terminal Task outcome when a retry sequence should count once.

A conversion rate over completed conversations, a contact rate over all attempts, and a final yield over Tasks answer different questions.

​

Add structured reporting fields

Reporting fields tell the Agent what to produce after a conversation. Use them for dispositions, qualification, appointment outcomes, escalation reasons, or values that standard log fields do not already contain.

• Give each field one purpose and a stable key.
• Prefer True / False, fixed choices, or numbers over free text.
• Define what the Agent should report when evidence is missing or inconclusive.
• Test positive, negative, and uncertain conversations.
• Do not rename keys after dashboards or integrations depend on them.

See Reporting for field setup.

​

Build metrics that can be explained

Every metric should identify:

• the records included and excluded
• the numerator and denominator
• the time range and grouping
• how missing values are treated
• the minimum useful sample size

Always review the event count with a rate. A high percentage based on two conversations is not a reliable trend. Validate a new metric against a bounded set of underlying logs before adding it to an organization dashboard. Organization metrics and Agent reporting fields are different settings: metrics describe how results are calculated, while reporting fields define what an Agent records.

​

Save versions before material changes

A named version preserves a known Agent configuration for comparison or recovery. Save one before changing core instructions, models, functions, integrations, routing behavior, or other settings that could materially change outcomes. Restoring a version prepares that configuration for review. Publish it only when you intend to make it live.

​

Run a controlled experiment

Use an experiment when live traffic should be distributed across two or more saved versions.

1. State one hypothesis and one primary metric.
2. Create the candidate versions.
3. Keep reporting definitions, audience, channel mix, and time period comparable.
4. Change one meaningful variable at a time when possible.
5. Run long enough to collect a useful sample.
6. Review the metric together with event count, failures, and guardrails.
7. End or pause the experiment before adopting the preferred version.

Do not declare a winner from a small sample or from variants that handled materially different traffic.

​

Use qualitative review too

Metrics tell you what changed; conversations help explain why. Review representative wins, failures, transfers, uncertain outcomes, and external actions before changing the Agent.