Chatbots are widely implemented across commercial, corporate, and government websites. Users often interact with chatbots without realizing they are communicating with software. In the not-so-distant future, most websites and communication platforms will likely include chatbot functionality. But what exactly is a chatbot?
A chatbot is a software service, powered by rules or artificial intelligence (AI), that is designed to simulate human interaction. Users access chatbots as stand-alone applications, via virtual assistants (Alexa, Siri), or through messaging apps such as Skype, Microsoft Teams, Facebook Messenger, Slack, or text messages. Chatbots are useful for answering frequent or routine questions. In business roles such as customer service and support, chatbots can result in significant cost savings. Chatbot applications, however, are virtually unlimited. Chatbots currently assist with weather services, news, finance, and even companionship.
This guide provides criteria to consider when designing a chatbot for your business. To create a successful chatbot implementation, you must (1) know your audience, (2) understand the key terms, and (3) understand the metrics by which to gauge your success.
1. Know Your Audience
Site users should:
• Be able to easily find what they are looking for.
• Receive guidance on questions and content searches.
• Be able to provide feedback if they can’t find what they are looking for.
• Be able to get relevant information based on the context and type of question.
• Be able to seamlessly reach a technician or agent to get answers if the chatbot is unable to answer questions effectively.
Content owners should:
• Be able to help users reach content easily by associating metadata tags (i.e., key phrases, IDs, or titles).
• Be assured that the chatbot will route users to the latest documents or pages.
• Be able to override questions, intents, or routes the chatbot has learned and apply new edits.
• Be assured that the chatbot will not route users to stale information (i.e., deprecated or deleted information).
• Be able to help users seamlessly reach a technician or agent to get answers if the chatbot is unable to answer questions effectively.
Site administrators should:
• Be able to evaluate the effectiveness of the chatbot using metrics such as system health, coverage, performance, and accuracy.
• Be able to start, stop, or retrain the chatbot on-demand through an easy-to-use admin module interface.
• Be assured that unauthorized users can’t change or influence the chatbot’s learning.
2. Understand Key Terms
The following are common chatbot terms users and administrators should be familiar with. Although knowledge of chatbot terms is not required for user interaction, businesses interested in chatbot implementations should understand how the terms relate to chatbot functionality.
• Utterance: Anything a user says to the chatbot.
o Example: If a user asks an online retail chatbot “How can I change my address for this account?” the entire sentence is an utterance.
• Intent: The action the user wants the chatbot to take.
o Example: If a user asks an online retail chatbot “How can I change my address for this account?” the intent is wanting to change the address associated with the user account.
o Intents are often given a name that combines a verb and a noun, such as “Change-Address.”
• Entity: The variable the user wants the bot to act upon. The entity modifies the intent.
o Example: If a user asks an online retail chatbot “How can I change my address for this account?” the entities are “address” and “account.”
 |
| Figure 1: Utterance, Intent, and Entity |
• Context: To determine a user’s intent, the chatbot must analyze the context of the conversation. To determine the context, the chatbot considers the page the user is on, what the user’s previous interactions with the chatbot were, and what questions the user previously asked the chatbot.
o In the example above, the context for the user’s question is the user’s account. The chatbot would determine the context by checking the account the user is logged into and by referencing accounts the user asked about in previous conversations.
• Content: The website, document, or any other text a chatbot uses for training. The content allows a chatbot to answer user questions with the correct context.
• Model: The AI component of a chatbot. The model consists of (1) the ability to understand natural language based on content, entities, and intents training, (2) the ability to evolve based on utterances and user interactions, and (3) the ability to construct sentences to respond to users.
• Training: The process of teaching a chatbot model new intents and questions as well as loading it with new responses and navigation destinations.
3. Use Metrics to Measure Chatbot Success
Businesses can measure results from chatbot implementations by examining four criteria: accuracy, performance, feedback, and adoption. It is important to continually track the results of chatbot implementations to refine the model and continually improve functionality.
A chatbot is accurate if:
• The chatbot has high confidence scores on natural language processing (NLP) platforms such as Microsoft’s Language Understanding Intelligent Service (LUIS). LUIS is a machine learning-based Microsoft cloud service that processes natural language text to predict overall meaning.
• The chatbot response rate is high. The response rate is the percentage of times the chatbot gives a meaningful response.
• The chatbot error rate is low. The error rate is the number of times a chatbot can’t find a meaningful response or the percentage of times it gives a generic response.
A chatbot meets performance goals if:
• Turnaround for business objectives is faster than before the chatbot implementation.
• The chatbot response to query ratio is high.
• The clickthrough rate (for links in chatbot responses) is high.
• The chatbot response time is better than the industry average.
A chatbot meets feedback goals if:
• The chatbot receives a high volume of positive-sentiment messages from users.
• The chatbot receives a high volume of thumbs up responses from users.
• Queries received by the chatbot per day.
• Active sessions per day.
• Active users per day.
• Returning users per day.
With careful planning, including knowing your audience, understanding key terms, and monitoring chatbot metrics, chatbots can be a significant asset to any business.