How to do Chatbot Analytics the Right Way

A chatbot is not a build and ship product; you do not release it into the wild and hope for the best. Chatbot development and delivery is an ongoing iterative process - for as long as the bot is live.

Every chatbot we develop ships with a bespoke reporting and analytics suite. We know the ability to monitor and analyse how people use your chatbot is critical to its success. When you have accurate and detailed data, you can prove the business case while optimising and improving engagement and results.

Benchmarking your chatbot (per industry)

Before we take a look at some key analytics, I want to touch on benchmarking briefly.

Benchmarking is something we recommend all clients do, it is the process of taking a snapshot of how your business is currently performing. We like clients to do this before the launch of their chatbot solution; it gives them accurate data for assessing the impact of the chatbot.

For a sales chatbot, some pre-deployment benchmarks might be:

• Daily/weekly/monthly revenue
• Daily/weekly/monthly inbound sales communications (calls, email, etc.)
• Sales staff workload assessment (number of staff, time spent on tasks, happiness in position, etc.)
• (If web-based chatbot) Website conversion rate, bounce rate and pages per visit
• Daily/weekly/monthly leads
• Daily/weekly/monthly sales

For a customer service chatbot:

• Daily/weekly/monthly inbound enquiries (calls, email, etc.)
• (If web-based chatbot) Website visitor frequency and recency (how often they come back and how much time has passed since last visit)
• Daily/weekly/monthly support tickets received
• Daily/weekly/monthly support tickets answered
• Support staff workload assessment (number of staff, time spent on tasks, happiness in position, etc.)

For an HR or internal communications chatbot:

• Daily/weekly/monthly enquiries to department
• HR staff workload assessment (number of staff, time spent on tasks, happiness in position, etc.)
• Average time to respond to enquiry
• Pre-deployment employee satisfaction survey

Ok, benchmarking done. Let's look at some key analytics you want your chatbot to track and improve.

Sidenote: these analytics are critical for all chatbots, regardless of their use or department.

Impact

Your chatbot has a goal, right? The single thing it needs to do? Well, we need to track that this goal is being met - the impact the chatbot is having.

For sales, it might be how many users are making a purchase and perhaps the average purchase price (for bonus points you can compare it to the benchmarked pre-chatbot number).

For customer service, it may be how many people said the chatbot response was correct or useful after being given an answer (your customer service chatbot is checking the user was satisfied, right?).

For HR, I would suggest it would be something similar to the customer service chatbot - a check to make sure the user was satisfied.

These impact analytics help make actionable tests and changes to your chatbot. It is only by improving the success of meeting the one true goal that your chatbot will start to anticipate the needs of users.

Power users

You know that rule in SaaS and ecommerce about analysing your best customers? Seeing where they clicked, what they read and their behaviour on their path to being your best customer? The same goes for a chatbot.

The people who talk with your chatbot the longest, who use it the most often and respond with the best feedback scores, these people are your power users. You should analyse and collect data on the interactions your users have to find your power users.

The goal is to tweak the chatbot's conversation flows and interactions to help everyone have a 'power user' experience. Sidenote: here's a fun read about Facebook realising their power users added seven friends within ten days. By setting up analytics to track all conversations, and comparing it to the one true goal above, you can work to ensure every conversation is successful.

Hot times

A pretty basic, but important, data point to track. When is your bot working the hardest? What are the times of the day when the most people are talking? This gives insight into who your users are, where they live and when they need the one true goal of your bot (to buy your stuff or to get your help).

This is useful for preventative messaging in the future. If you know when most of your users are buying or asking for help, you understand the best time to contact them preemptively. Critical for future marketing or sales campaigns and "you might need help" type email or communication. It will also help the dev/product teams to know when to push updates.

Sentiment

Warning: very few (if any?) drag-and-drop and off-the-shelf chatbot solutions measure this. If it is important to you, then make sure it is available before choosing a vendor.

By analysing and measuring the sentiment of inbound messages, you basically have an instant pulse of your consumers. You can tell what they feel and think about your company, your services and products.

This sentiment analysis will be useful across many business units. Here are a few top-of-head examples:

• Marketing to assess the impact of their marcomms and campaigns
• Sales to differentiate hot from cold leads
• Customer service to know what is annoying customers and what support content to create
• Product managers to know which features are working
• Developers to spot and understand bugs and application errors
• Brand managers to see what makes users happy/unhappy and assess overall brand sentiment

Improve usefulness

By measuring core metrics like retention and if a user uses the chatbot without switching to another channel, you can see whether your chatbot is doing its job and providing value. Alongside this, use analytics to track errors and fallbacks-to-human. You can find out which sections of conversation, or functions, where users are getting stuck.

By combining and improving the core metrics with the error and fallback metrics, you make your chatbot more useful over time. We use these metrics to ensure the chatbot is doing its job and meeting the one true goal better than any other channel can.

Speed to goal

Another obvious but frequently skipped metric is the "how quick the user got to where they are going" metric. If your chatbot is designed to sell, how quickly did it sell? If it aims to help, how quickly did it help?

Your users do not want to type War and Peace in a conversation with your chatbot. Our research shows they want to ask a question and get an instant response (or if sales they want accurate and relevant recommendations). Simply, making your chatbot help someone quicker makes users happier.

The importance of analytics

Analytics gives you the power to A/B test, to improve and develop your chatbot. If you do it right, a chatbot gives you unparalleled insight into your audience. You will be swimming in conversation data, sentiment analysis and actionable intelligence. In fact, I can guarantee that soon after launch; you will have so much data and insight you will not know what to do with it (we are here to help).

Use analytics to test different conversation flows, different features and goals of the chatbot. Record, measure and understand all the data your analytics create - do not pivot and adapt based on hunches or personal feelings and thoughts.

Always, always, always use accurate and unbiased data.

---