Compare NLP Engines: Wit.ai, Lex, API.ai, Luis.ai, Watson Assistant

Chatbot popularity is growing over time. Still, this new means of communication between customers and brands seems promising and unclear at the same time.

On one hand, what could be better than a simple dialog between a human and a chatbot able to memorize things, perform complicated calculations, and make API calls at the same time? On the other hand, creating a bot with this level of complexity that would stay neutral and understand user needs doesn’t seem simple at all. Fortunately, there are tools to help.

Although it sounds familiar, Natural Language Processing (NLP) is usually associated with something complicated. And in fact, creating an NLP engine from scratch requires significant efforts.

However, today it is not always necessary. There are many existing NLP engines that help developers empower their bots with text or voice processing technology.

What Is an NLP Engine

The main purpose of natural language processing is to understand user input and translate it into computer language. To make it possible, developers teach a bot to extract valuable information from a sentence, typed or pronounced, and transform it into a piece of structured data. That’s what NLP engines do.

As soon as user query becomes clear, the program that uses NLP engine – chatbot in this case – will be able to apply its logic to further reply to the query and help users achieve their goals.

How to Benefit from Using NLP Engines

NLP engines use human language corpus to extract the meaning of user requests and understand common phrases.

Here is a simple example. Let’s say you are building a restaurant bot and you want it to understand user request to book a table.

Compare these sentences:

• Can you make a reservation at Red Lobster tonight?
• Could you book a restaurant in Manhattan?
• Could you book a restaurant near me?
• Can you reserve a table for 2 persons at Red Lobster?
• How can I make a dining reservation?
• Could you help me book a table at Red Lobster, please?

These sentences are clear for a human who understands that these user queries are similar. However, they look completely different to a machine.

Luckily, using NLP we can teach the machines to see these similarities. In fact, we have already taught some popular engines how to do it.

For example, an NLP engine knows that phrases like “can you”, “how can I”, “could you help me” are general. They are synonymous and don’t have much influence on the semantics. NLP engines tend to ignore these “senseless” parts when they extract the meaning.

Today, this benefit cuts down on the need to create an NLP engine in house from scratch and teach it to understand natural language from the very beginning. Moreover, some available engines have a prebuilt model for domain. So teaching an engine to understand a domain specific language is easier too.

Glossary

Before getting down to engine comparison, let’s go through the glossary of terms that NLP engineers use:

• Intent. Intent is the core concept of building conversational UI. In short, Intent is the task user wants to accomplish or the problem user wants to solve.
• Utterance. By Utterances (also Stories) NLP engines mean the example of sentences user may type or tell when he or she refers to a specific Intent.
• Entity. Entities (also Parameters and Slots) include the details important for the user’s Intent. It can be anything: location, date, time, and even cuisine type.
• Context. Context helps to save and share the Parameters across the whole Session.
• Session. Session is one conversation from the beginning to the end.

The image below explains these NLP terms using visualization:

These are 3 important tasks that existing NLP tools can do:

• They provide the corpus of natural language. It simplifies development and helps to focus on domain specific language.
• They help to extract the user’s Intents, Parameters (Entities), maintain Context and save the states of Parameters across the whole Session.
• They assign the actions depending on Context and call your API to fulfill user requests.

Now let’s review what kind of NLP engines/tools are available in the market and what capabilities they have.

Wit.ai

Wit.ai was acquired by Facebook in 2015, and since then has been actively developed by the company.

Training

The training of this engine goes around Stories (domain specific use cases). The tool learns conversation flows from the examples of user input and chatbot responses. As any other NLP engine, it allows to understand user input after certain training, identify Intent, extract Entities, and predict what your bot should do based on the current Context and user query.

You can train your model via web interface. The system validates every sentence against an existing model and suggests to highlight the prebuilt Entities, if any. Of course, you can define your own Entities/Parameters and set possible values for all of them. Standard Parameters include location, temperature, date and time, age, URLs, emails, reminders, distance, etc.

Features

In general, it’s easy to get started. API is well documented. Before taking off, you can browse through the existing bots and apps built on wit.ai: many of them are open to the public.

You can use them not only for inspirational purposes, but also to jumpstart your project. Platform allows to copy other developers’ Stories together with their training.

Wit.ai has a visual chat UI for testing conversations where you can see the steps that systems recognize. It helps during both bot creation and model improving.

Language and cost

Platform supports about 50 different languages and is completely free of charge.

Final word

Wit.ai is a part of Facebook infrastructure. It makes it a prefect choice for those who plan to develop chatbots for Facebook Messenger. Because of good user interface and straightforward documentation starting a project using this platform is easy. In short, it appears a good option for simple B2C bots and various MVP projects.

API.ai

API.ai is a popular platform for building conversational interfaces. Before Google bought it in December 2016, the platform belonged to an independent development company.

Training

As any other NLP engine, its functionality allows to train the model around a specific user Intent. Apart from that, bot and app developers can benefit from using prebuilt models.

API.ai offers 33 prebuilt Agents that you can import to your project and customize depending on your needs. Among the list of prebuilt Agents you will find many common ones, such as “Navigation”, “Hotel Booking”, “Small Talk”, “Translator”, “Weather”, “News”, etc.

If your chatbot is supposed to be fully unique in terms of dialogs, you don’t need most of these agents. However, you can still import “Small Talk” module with common topics like “hello,” “goodbye,” “emotions,” or “courtesy,” and make your bot polite ?

If you want to train your model manually, you can either put your phrases via web UI or import the model from .txt or .zip file. Then you’ll go through the typical for NLP engines process: define Intents, Entities and programming Actions that bots should do.

Features

API.ai provides one-click integrations with such platforms as Facebook Messenger, Slack, Viber, Kik, Telegram, Amazon Alexa, Line, Microsoft Cortana. It helps to develop your bot for many platforms at once. Also, it has a set of SDKs for Node.js, Ruby, Android, iOS, C#, Python, etc., so you can use it not only for chatbots, but applications and devices.

It is worth to mention that API.ai supports voice interfaces. It allows you to build the Agent that understands text and voice without additional efforts. Later, when you test your Agent you can test both text and vocal dialogs.

Language and cost

At the moment, the platform supports 15 natural languages. Though we can expect the number of natural languages, prebuilt models, and integrations to grow over time. Just like wit.ai platform, API.ai is completely free.

Final word

The platform is well-documented and has a friendly user interface. Considering the number of prebuilt agents, it is really easy to start building a chatbot that fits many platforms at once. Moreover, it’s a good engine to build simple or middle level chatbots or virtual assistants with voice interface.

*Api.ai has been changed to dialogflow.com

Lex

Recently, Amazon has announced the launch of its Lex services built on Amazon Alexa Skill Kit. It’s a truly exciting news, since Alexa has already been tested and trained by more than 3 million Amazon Alexa users.

Training

Lex NLP engine has a structure similar to other engines. You have to build conversations around the user Intent and define your Utterances and their Slots. The Utterances don’t have to match exactly with what the user says. Though it’s recommended to define Utterances with different Slot variations. Each Slot can be associated with one or more Prompts that Amazon Lex uses to get value from the user’s input.

The number of system slot type references is imposing. If it’s relevant for the Slot nature, you can assign the card image to the Prompt. As a result, it’ll give a user visual clues of possible options. In other words, using Lex web interface you can build conversational interfaces using both simple text and cards with images and buttons.

Features

Lex web interface gives you not only configuration features, but also a monitoring dashboard where you can review different metrics such as Text Request (Count), Speech Request (Count), Text Missed Utterances (Count), Speech Missed Utterances (Count), etc.

As expected, Amazon Lex has tight integration to different Amazon services such as Lambda, Dynamo DB, SNS/SES, and others. In addition, the developers have provided connectors for popular SaaS applications like Salesforce, Microsoft Dynamics, Marketo, Zendesk, QuickBooks, and Hubspot.

Today, Amazon Lex is available in a preview mode in the US East (Northern Virginia) region. Surely, it’ll increase the list of regions, integrations, supported messengers, and prebuilt intents in the nearest future.

Language and cost

Unlike the wit.ai and API.ai, Amazon Lex is a paid service. According to Amazon, “you can make 10,000 text requests and 5,000 speech requests each month at no charge for the first year. After that you will pay $4.00 for each 1,000 speech requests and $0.75 for every 1,000 text requests.”

Final word

It’s a bit too early to make any opinion on Lex because the system is in the preview mode. Although, it’s definitely worth attention. First, the platform is built on top of Amazon Alexa Skill Set, so it’s familiar. Secondly, it’s reliable, since, as a rule, Amazon develops reliable and mature cloud services for users.

LUIS.ai

LUIS.ai is Microsoft Language Understanding Intelligent Service that was introduced by Microsoft in 2016. Nowadays, Microsoft provides several useful tools for bot makers. Besides LUIS NLP engine, tech giant offers Microsoft Bot Framework and Skype Developer Platform.

Microsoft Bot framework helps to build, test, and deploy bots for many well-known platforms such as Facebook, Skype, Slack, Cortana, Kik, Telegram, and SMS. Skype Developer Program, in turn, gives the opportunity to build apps for Skype.

LUIS.ai can be applied both with or without Microsoft Bot Framework. However, the combination seems to be a more reasonable option.

Training

As in the previous cases, to test and train your model and build an NLP-driven bot you should configure your Intents and Entities. Additionally, there are some prebuilt domains that you can import to your chatbot together with its Entities, Intents, and Utterances.

The following domains are available at the moment: Calendar, Camera, Communication, Entertainment, Events, Fitness, Gaming, HomeAutomation, MovieTickets, Music, Note, OnDevice, Places, Reminder, RestaurantReservation, Taxi, Translate, TV, Utilities, Weather, and Web.

If you don’t need the entire list of Intents and Entities from the mentioned domains, you can import specific Intents (around 170 available up to this point) and/or import specific Entities. The list of default Utterances isn’t that ample though, so it makes sense to add additional ones for better prediction.

As soon as you configure Intents, add Utterances, and define Entities, you can start training your model. LUIS.ai provides a handy interface that shows you the predicted interpretation of the Utterance and extracted Entities and Intents.

Features

This tool is abile to configure Features. It helps to improve model and Intents detection. All you need is to add a “Phrase List” or/and “Patterned Feature.” Phrase list allows to define the list of similar objects (for example, names of cities or cuisine types), while the “Patterned Feature” enables to define a regular expression which helps your bot to recognize domain specific patterns (for example, flight numbers).

Language and cost

There are 10 natural languages available. In terms of cost, you can make use of 10,000 transactions for free each month, then it’ll cost you $0.75 per 1,000 transactions.

Final word

Microsoft LUIS is a good option for .NET developers and bot projects that require integration with enterprise software. It’s a good fit for Cortana functionality, IoT applications, and virtual assistant apps.

Watson Assistant

There is one more well-known player in the market. Actually, one of the oldest. IBM provides its Watson Assistant tool, IBM Watson, that also works as a good fit for bot creation.

Training

As other NLP tools, it provides you with a web interface for defining Intents and Entities. You can import and export Intents as well as define what type of phrases user says when he or she is talking about a specific Intent. Defining your Entities, you can list all values and determine the synonyms relevant for a specific value.

Furthermore, you can play with Watson’s Dialog interface to build a tree of conversation flow. To start, you will need to create a dialog branch for each Intent and then set a condition based on the Entities in the input.

Of course, you are able to test your model to improve it before publishing your bot or app. The drawback is the lack of prebuilt Entities that you could import to your project.

Update: IBM has recently added a plethora of prebuilt intents that focus on customer care, industry and other common chatbot controls. Users can add these intents to their virtual assistants, modify them, or add their own utterances.

Features

This platform is powerful, but doesn’t seem friendly enough. On one hand, there are many building blocks that you can use in your application in addition to the Dialog API available in the Watson Assistant interface. On the other hand, you’ll have to spend much time to integrate them into your project.

Language and cost

Watson Assistant tool supports 10 natural languages. It’s a costly solution; you’ll pay $0.02 per call, but for an enterprise-level bot with a proven business model this price is not such a big deal.

Final word

Watson Assistant tool requires some effort to start working with it and take advantage of its integrations. It’s an enterprise level solution, and it doesn’t sound like an option for an MVP chatbot project.

Conclusion

There is a number of good engines in the market that can help you start the bot quickly. These tools have just started shaping up, but they improve to become better and better.

Evidently, all well-known market players are in the game. In fact, all NLP APIs listed in this post are either acquired by Amazon, Google, Microsoft, Facebook, IBM or appear as the original products of these brands. Recently, Apple has joined the club and announced its own NLP engine.

There’s no doubt, these tools have area for improvements, since developers do experience some issues working with these platforms. For example, these APIs can learn only from examples and fail to provide options to take advantage of additional domain knowledge. Some developers complain about the accuracy of algorithms and expect better tools for dialog optimization.

Moreover, some of platform features such as Stories in Wit.ai or Training in Api.ai are still in beta. However, it’s just the beginning, and we should keep that in mind. The more conversational interfaces are created, the better results NLP engines will generate.

See the features and details of top NLP tools for chatbot creators compared in the table below:

*Api.ai has been changed to dialogflow.com

**Watson Conversation has been changed to Watson Assistant, dozens of customizable prebuilt intents have been added, so as integration capabilities.

Would you like to develop a smart AI chatbot for your business?  Contact our team to talk about your chatbot ideas, create a chatbot using an NLP engine, or hire a chatbot developer to develop a custom chatbot strategy for your business.