AI lead qualification pilot metrics: review decisions before counting wins

An AI lead qualification pilot needs a way to inspect decisions before the team treats a busy handoff queue as success. A booked meeting tells you that someone booked. It does not tell you whether the qualification decision matched your team's requirements or whether suitable inquiries were missed.

Use this review method to evaluate one configured workflow. Datagran connects lead conversations, human handoffs, CRM updates, and follow-up. The worksheet below is an operating method you can adapt, not a claim that every metric is a built-in report or that a pilot will increase revenue. Every number in the worked example is fictional.

Define the decision and the review cohort

Write the qualification rubric before reviewing the agent's decisions. State what evidence makes an inquiry suitable for the next sales step, what evidence makes it unsuitable, and what must remain unresolved. Use the human handoff guide for qualification questions and the context a salesperson needs.

A hypothetical rubric might require a stated business need, a supported use case, and interest in a sales conversation. If an essential integration has not been named or checked, the reviewer cannot mark that requirement confirmed. A missing budget answer is unknown information, not an automatic rejection.

Then record:

• The intake channel, observation period, timezone, and review cutoff.
• The unit being counted, such as one inquiry about one request.
• The workflow, configuration, and rubric version.
• Which inquiries are included and why any are excluded.
• The person responsible for reviewing disagreements and incomplete evidence.

Three messages about the same request should not silently become three independently qualified leads. Keep inquiry matching separate from contact matching, using the duplicate-lead workflow when identity is uncertain.

Review both positive and negative decisions

For this exercise, a positive decision means that the inquiry meets the written qualification rubric. It does not mean that the person booked, bought, or paid.

Have a reviewer inspect the underlying conversation against that rubric. Where practical, make the initial review without showing the agent's label or a later booking outcome. Retain the source reference and the reason for the reviewer's label. A second review can resolve a disagreement; unresolved cases need their own category.

Review inquiries the agent marked unsuitable as well as the ones it handed to sales. Reviewing only generated handoffs can reveal false positives while hiding false negatives. If you sample rather than inspect the full cohort, state the selection method and the reviewed counts. A sample chosen for known failures cannot be presented as the full intake's error rate.

Freeze the original rubric and labels for that review. If the team changes its requirements after seeing the results, evaluate the revised rule separately. Quietly relabeling the earlier cohort makes the comparison impossible to interpret.

Keep a review worksheet with the source evidence

Use one row per declared unit of analysis, with these fields:

• Inquiry reference and the time it entered the cohort.
• Workflow and rubric version used for the decision.
• Agent decision and the evidence it cited, if available.
• Reviewer label: meets the rubric, does not meet it, or unresolved.
• Conversation reference supporting that label.
• Missing evidence, disagreement, or exclusion reason.
• Reviewer, review time, and next action.

Keep private conversation contents in the authorized system. An aggregate pilot report can use non-identifying references and counts. The operating worksheet should not become a public export of prospect details.

A reviewer label is a reference decision under the declared rubric. It can still be wrong or incomplete. Preserve disagreements so another person can inspect them instead of treating a human label as unquestionable truth.

Work through a hypothetical confusion matrix

Suppose a fictional pilot receives 25 inquiries. Reviewers reach a label for 20. The other five are unreviewed or unresolved and stay outside the classification calculations.

Among the twenty labeled inquiries:

• Six meet the rubric and the agent marks them positive. These are true positives.
• Four do not meet the rubric but the agent marks them positive. These are false positives.
• Two meet the rubric but the agent marks them negative. These are false negatives.
• Eight do not meet the rubric and the agent marks them negative. These are true negatives.

The four groups reconcile: 6 + 4 + 2 + 8 = 20. The agent predicted ten positives, while reviewers identified eight reference positives.

Precision asks how many predicted positives meet the rubric: 6 divided by 10, or 60%. Recall asks how many reference positives the agent found: 6 divided by 8, or 75%. Accuracy counts correct positive and negative decisions: 14 divided by 20, or 70%.

Review coverage is a different fraction: 20 divided by 25, or 80%. Report the five unresolved cases beside the other counts. Do not convert missing labels into negatives or assume that those five cases behave like the twenty labeled ones.

Google's classification guide defines precision, recall, and accuracy and explains why accuracy can be misleading when classes are imbalanced. The example here applies those definitions to a proposed qualification review. It does not describe Datagran campaign performance.

Handle zero denominators explicitly

If the agent predicts no positive cases, precision has a zero denominator. If the reviewed cohort has no reference positives, recall has a zero denominator. Leave the corresponding metric undefined and show the counts that explain it. An undefined fraction is not evidence of 100% performance.

Retain the decision rule used for each run. If your configured workflow uses a score threshold, a new threshold creates a different set of decisions. A model's own confidence score is not an independent reviewer label or proof that an integration, booking, or CRM write succeeded.

Inspect the errors that matter to the team

The same fraction can hide very different failures. Read the false-positive and false-negative cases and record a useful reason, such as an unsupported integration assumption, a support request routed to sales, missing source history, or an unrecognized request for human help.

A missed suitable inquiry may need prompt follow-up. A false positive may waste a salesperson's time or send a prospect to the wrong team. Decide which error types require a pause or immediate review before expanding the workflow. A small illustrative sample cannot establish that a workflow is safe, statistically reliable, or causally better than another process.

If you compare versions, keep the rubric, intake definition, observation window, and review method comparable. Show changes in channel mix or unresolved coverage that could explain a difference. Keep tuning cases separate from a later review set, and do not report the same repeatedly adjusted sample as fresh validation.

Track downstream outcomes separately

After inspecting qualification decisions, follow the relevant records through the next steps. Keep these outcomes distinct:

• A handoff with a named owner.
• A handoff that the owner accepted.
• A confirmed demo booking and, separately, an attended meeting.
• A qualified opportunity under the team's opportunity definition.
• A won deal and a recorded customer payment.

Name the system and evidence for each event, plus its observation cutoff. An outcome can occur after the classification review ends. Leave it pending or unknown until its evidence is available. A missed response is a separate operating question covered by the speed-to-lead guide. Campaign credit and collected revenue belong in the attribution checklist.

Check the current product guides for the sources, connections, and approvals your Datagran setup supports. An enabled connection does not imply that every CRM exposes the same stage triggers or that all events reach one report automatically.

Decide what to change before expanding the pilot

Use a short release review:

1. Confirm that every included inquiry belongs to the declared cohort and appears once.
2. Reconcile the four decision groups, unresolved cases, and exclusions to their source records.
3. Show denominators and review coverage alongside the metrics.
4. Assign an owner and next action to the material errors and disagreements.
5. Verify downstream events in their actual systems, allowing for observation lag.
6. Record whether the team will keep the current scope, revise the workflow, pause it, or expand it.

Keep the result tied to the evidence the team reviewed. A useful pilot gives you a reproducible decision process and a clear next change. Revenue claims require separate confirmed business outcomes.