Chatbot metrics that mean something, and three that mislead
Containment rate, message volume and deflection are the numbers most dashboards lead with, and all three can improve while service gets worse. Here is what to watch instead.
Key takeaways
- Containment, deflection and message volume can all improve while service gets worse.
- Repeat contact within forty-eight hours is the most honest measure of whether conversations were resolved.
- Cost per resolved conversation, including Meta charges and agent time, is what makes the business case.
- Read twenty transcripts a week; recurring failures are obvious in transcripts and invisible in aggregates.
The most useful chatbot metrics are resolution rate confirmed by the customer, repeat contact within forty-eight hours, time to first human reply after escalation, and satisfaction split by whether the conversation escalated. The metrics most dashboards lead with, containment, deflection and total messages handled, can all improve while the customer experience deteriorates.
Three that mislead
- Containment rate. The share of conversations closed without a human. A bot that makes escalation difficult scores highly here and serves people badly. Useful only when read alongside repeat contact.
- Deflection. Usually counts conversations that did not reach an agent, which includes every customer who gave up. Abandonment and deflection look identical in most reporting.
- Messages handled. Grows when the bot is verbose or when customers have to rephrase repeatedly. A rising message count per conversation is more often a symptom than an achievement.
Four that hold up
Confirmed resolution asks the customer whether the issue was solved and records the answer. It is the only direct measure, and the cost is one extra message. Repeat contact within forty-eight hours catches the conversations that were closed rather than resolved, and it is the single most honest number in conversational support. Time to first human reply after escalation measures the handover, which is where trust is won or lost. And satisfaction split by escalation tells you whether the automation boundary is in the right place.
The commercial layer
For anything revenue-facing, track conversion from conversation, average order value from conversations against other channels, and cost per resolved conversation including both the Meta conversation charge and the agent time. Cost per resolved conversation is the figure that makes the business case, because it can be compared directly against the cost of handling the same volume by phone.
Include the block and report rate in the same report. It is the leading indicator of a channel problem, and because a falling quality rating reduces how many customers you may message, it belongs next to the commercial numbers rather than buried in a platform console.
Read transcripts, not only dashboards
Sample twenty conversations a week and read them end to end. Every recurring failure we have found in a deployed bot was obvious within ten transcripts and invisible in the aggregate metrics, because a bot that fails politely produces a normal-looking dashboard. Rotate who reads them, and include someone who was not involved in building it.
Reading the numbers by segment
Aggregate figures hide the failures that matter. Split every metric by language, because an Arabic pass rate well below the English one is common and invisible in the total. Split by conversation topic, since a bot that handles order status excellently and returns badly will look adequate overall. And split by whether the customer was new or returning, because returning customers know how to phrase things and mask problems new ones hit immediately.
Setting a baseline before launch
Record what the equivalent human-handled conversations cost and how long they took before the automation goes live. Without that baseline, every later claim about savings is an assertion, and the first time someone challenges the business case you will have no answer. A fortnight of manual measurement beforehand is cheap and settles the argument permanently.
Abandonment and deflection look identical in most reporting. Only one of them is a success.