How To Create an Intelligent Chatbot in Python Using the spaCy NLP Library

The author selected the COVID-19 Relief Fund to receive a donation as part of the Write for DOnations program.

Introduction

Interacting with software can be a daunting task in cases where there are a lot of features. In some cases, performing similar actions requires repeating steps, like navigating menus or filling forms each time an action is performed. Chatbots are virtual assistants that help users of a software system access information or perform actions without having to go through long processes. Many of these assistants are conversational, and that provides a more natural way to interact with the system.

To create a conversational chatbot, you could use platforms like Dialogflow that help you design chatbots at a high level. Or, you can build one yourself using a library like spaCy, which is a fast and robust Python-based natural language processing (NLP) library. spaCy provides helpful features like determining the parts of speech that words belong to in a statement, finding how similar two statements are in meaning, and so on.

In this tutorial, you will create a chatbot that not only helps users simplify their interactions with a software system, but is also intelligent enough to communicate with the user in natural language (American English in this tutorial). The chatbot will use the OpenWeather API to tell the user what the current weather is in any city of the world, but you can implement your chatbot to handle a use case with another API.

Prerequisites

Before you begin, you will need the following:

• Python 3 installed and a programming environment set up. You can complete this for your machine with one of the How To Install Python 3 and Set Up a Local Programming Environment tutorials.
• An API key for OpenWeather. Visit the OpenWeather website to create an account. Make sure to confirm your email address. After registering successfully, visit the API keys page to view the API key automatically created for your account. This key should be an alphanumeric sequence of characters.

This tutorial assumes you are already familiar with Python—if you would like to improve your knowledge of Python, check out our How To Code in Python 3 series. This tutorial does not require foreknowledge of natural language processing.

Step 1 — Setting Up Your Environment

In this step, you will install the spaCy library that will help your chatbot understand the user’s sentences.

Having set up Python following the Prerequisites, you’ll have a virtual environment. Let’s activate that environment.

Make sure you are in the directory where you set up your environment and then run the following command:

source my_env/bin/activate

Now install spaCy:

pip install -U spacy

Finally, you will download a language model. spaCy’s language models are pre-trained NLP models that you can use to process statements to extract meaning. You’ll be working with the English language model, so you’ll download that.

Run the following command:

python -m spacy download en_core_web_md

If you run into an error like the following:

Output
ERROR: Failed building wheel for en-core-web-md

You need to install wheel:

pip install -U wheel

Then download the English-language model again.

To confirm that you have spaCy installed properly, open the Python interpreter:

python

Next, import spaCy and load the English-language model:

>>> import spacy
>>> nlp = spacy.load("en_core_web_md")

If those two statements execute without any errors, then you have spaCy installed.

Now close the Python interpreter:

>>> exit()

You now have everything needed to begin working on the chatbot. In the next section, you’ll create a script to query the OpenWeather API for the current weather in a city.

Step 2 — Creating the City Weather Program

In this section, you will create a script that accepts a city name from the user, queries the OpenWeather API for the current weather in that city, and displays the response.

First, create and open a Python file called weather_bot.py with your preferred editor:

nano weather_bot.py

Next, you’ll create a function to get the current weather in a city from the OpenWeather API. This function will take the city name as a parameter and return the weather description of the city.

Add the following code into your weather_bot.py file:

import requests

api_key = "your_api_key"

def get_weather(city_name):
	api_url = "http://api.openweathermap.org/data/2.5/weather?q={}&appid={}".format(city_name, api_key)

	response = requests.get(api_url)
	response_dict = response.json()

    weather = response_dict["weather"][0]["description"]

	if response.status_code == 200:
		return weather
	else:
		print('[!] HTTP {0} calling [{1}]'.format(response.status_code, api_url))
		return None

First, you import the requests library, so you are able to work with and make HTTP requests. Make sure to replace your_api_key with your own API key. The next line begins the definition of the function get_weather() to retrieve the weather of the specified city.

In this function, you construct the URL for the OpenWeather API. This URL returns the weather information (temperature, weather description, humidity, and so on) of the city and provides the result in JSON format. After that, you make a GET request to the API endpoint, store the result in a response variable, and then convert the response to a Python dictionary for easier access.

On the next line, you extract just the weather description into a weather variable and then ensure that the status code of the API response is 200 (meaning there were no issues with the request). Finally, you return the weather description.

If there is an issue with the request, the status code is printed out to the console, and you return None.

To test the script, call the get_weather() function with a city of your choice (for example, London) and print the result. Add the highlighted code following your function:

import requests

def get_weather(city_name):

  ...

  return weather

weather = get_weather("London")
print(weather)

Save and run the script:

python weather_bot.py

You will receive a result like the following:

Output
scattered clouds

Having completed that successfully, you can now delete the last two lines from the script.

Open it with:

nano weather_bot.py

Then delete the two highlighted lines at the end of the file:

import requests

def get_weather(city_name):

  ...

  return weather

weather = get_weather("London")
print(weather)

Save and close the file.

You now have a function that returns the weather description for a particular city.

In the next step, you’ll create a chatbot capable of figuring out whether the user wants to get the current weather in a city, and if so, the chatbot will use the get_weather() function to respond appropriately.

Step 3 — Creating the Chatbot

In the previous two steps, you installed spaCy and created a function for getting the weather in a specific city. Now, you will create a chatbot to interact with a user in natural language using the weather_bot.py script.

You’ll write a chatbot() function that compares the user’s statement with a statement that represents checking the weather in a city. To make this comparison, you will use the spaCy similarity() method. This method computes the semantic similarity of two statements, that is, how similar they are in meaning. This will help you determine if the user is trying to check the weather or not.

To begin, open the script:

nano weather_bot.py

Then, import spaCy and load the English language model:

import spacy
import requests

nlp = spacy.load("en_core_web_md")

. . .

After the get_weather() function in your file, create a chatbot() function representing the chatbot that will accept a user’s statement and return a response.

Following your definition, add the highlighted code to create tokens for the two statements you’ll be comparing. Tokens are the different meaningful segments of a statement, like words and punctuation. This is necessary to allow spaCy to compute the semantic similarity:

import spacy

. . .

def chatbot(statement):
  weather = nlp("Current weather in a city")
  statement = nlp(statement)

Here the weather and statement variables contain spaCy tokens as a result of passing each corresponding string to the nlp() function.

Save and close your file.

Next you’ll be introducing the spaCy similarity() method to your chatbot() function. The similarity() method computes the semantic similarity of two statements as a value between 0 and 1, where a higher number means a greater similarity. You need to specify a minimum value that the similarity must have in order to be confident the user wants to check the weather.

For example, if you check the similarity of statements 2 and 3 with statement 1 following, you get:

1. Current weather in a city
2. What is the weather in London? (similarity = 0.86)
3. Peanut butter and jelly (similarity = 0.31)

To try this for yourself, open the Python interpreter:

python

Next, import spaCy and load the English-language model:

>>> import spacy
>>> nlp = spacy.load("en_core_web_md")

Now let’s create tokens from statements 1 and 2:

>>> statement1 = nlp("Current weather in a city")
>>> statement2 = nlp("What is the weather in London?")

Finally, let’s obtain the semantic similarity of the two statements:

>>> print(statement1.similarity(statement2))

You will receive a result like this:

Output
0.8557684354027663

Setting a low minimum value (for example, 0.1) will cause the chatbot to misinterpret the user by taking statements (like statement 3) as similar to statement 1, which is incorrect. Setting a minimum value that’s too high (like 0.9) will exclude some statements that are actually similar to statement 1, such as statement 2.

We will arbitrarily choose 0.75 for the sake of this tutorial, but you may want to test different values when working on your project.

Let’s add this value to the script. First, open the file:

nano weather_bot.py

Then add the following highlighted code to introduce the minimum value:

import spacy

. . .

def chatbot(statement):
  weather = nlp("Current weather in a city")
  statement = nlp(statement)
  min_similarity = 0.75

Now check if the similarity of the user’s statement to the statement about the weather is greater than or equal to the minimum similarity value you specified. Add the following highlighted if statement to check this:

import spacy

. . .

def chatbot(statement):
  weather = nlp("Current weather in a city")
  statement = nlp(statement)
  min_similarity = 0.75

  if weather.similarity(statement) >= min_similarity:
    pass

The final step is to extract the city from the user’s statement so you can pass it to the get_weather() function to retrieve the weather from the API call. Add the following highlighted for loop to implement this:

import spacy

...

def chatbot(statement):
  weather = nlp("Current weather in a city")
  statement = nlp(statement)
  min_similarity = 0.75

  if weather.similarity(statement) >= min_similarity:
    for ent in statement.ents:
      if ent.label_ == "GPE": # GeoPolitical Entity
        city = ent.text
        break

To do this, you’re using spaCy’s named entity recognition feature. A named entity is a real-world noun that has a name, like a person, or in our case, a city. You want to extract the name of the city from the user’s statement.

To extract the city name, you get all the named entities in the user’s statement and check which of them is a geopolitical entity (country, state, city). To do this, you loop through all the entities spaCy has extracted from the statement in the ents property, then check whether the entity label (or class) is “GPE” representing Geo-Political Entity. If it is, then you save the name of the entity (its text) in a variable called city.

You also need to catch cases where no city was entered by adding an else block:

import spacy

...

def chatbot(statement):
  weather = nlp("Current weather in a city")
  statement = nlp(statement)
  min_similarity = 0.75

  if weather.similarity(statement) >= min_similarity:
    for ent in statement.ents:
      if ent.label_ == "GPE": # GeoPolitical Entity
        city = ent.text
        break
      else:
        return "You need to tell me a city to check."

Now that you have the city, you can call the get_weather() function:

import spacy

...

def chatbot(statement):
  weather = nlp("Current weather in a city")
  statement = nlp(statement)
  min_similarity = 0.75

  if weather.similarity(statement) >= min_similarity:
    for ent in statement.ents:
      if ent.label_ == "GPE": # GeoPolitical Entity
        city = ent.text
        break
      else:
        return "You need to tell me a city to check."

    city_weather = get_weather(city)
    if city_weather is not None:
      return "In " + city + ", the current weather is: " + city_weather
    else:
      return "Something went wrong."
  else:
    return "Sorry I don't understand that. Please rephrase your statement."

Recall that if an error is returned by the OpenWeather API, you print the error code to the terminal, and the get_weather() function returns None. In this code, you first check whether the get_weather() function returns None. If it doesn’t, then you return the weather of the city, but if it does, then you return a string saying something went wrong. The final else block is to handle the case where the user’s statement’s similarity value does not reach the threshold value. In such a case, you ask the user to rephrase their statement.

Having completed all of that, you now have a chatbot capable of telling a user conversationally what the weather is in a city. The difference between this bot and rule-based chatbots is that the user does not have to enter the same statement every time. Instead, they can phrase their request in different ways and even make typos, but the chatbot would still be able to understand them due to spaCy’s NLP features.

Let’s test the bot. Call the chatbot() function and pass in a statement asking what the weather is in a city, for example:

import spacy

. . .

def chatbot(statement):

. . .

response = chatbot("Is it going to rain in Rome today?")
print(response)

Save and close the file, then run the script in your terminal:

python3 weather_bot.py

You will receive output similar to the following:

Output
In Rome, the current weather is: clear sky

You have successfully created an intelligent chatbot capable of responding to dynamic user requests. You can try out more examples to discover the full capabilities of the bot. To do this, you can get other API endpoints from OpenWeather and other sources. Another way to extend the chatbot is to make it capable of responding to more user requests. For this, you could compare the user’s statement with more than one option and find which has the highest semantic similarity.

Conclusion

You have created a chatbot that is intelligent enough to respond to a user’s statement—even when the user phrases their statement in different ways. The chatbot uses the OpenWeather API to get the current weather in a city specified by the user.

To further improve the chatbot, you can:

• Check the OpenWeather API guide for additional weather functions you can add.
• Visit the spaCy website to see other features you can implement to make the chatbot more intelligent.
• Improve your knowledge of natural language processing with our tutorial on How To Work with Language Data in Python 3 using the Natural Language Toolkit (NLTK) or try sentiment analysis in How To Perform Sentiment Analysis in Python 3 Using the Natural Language Toolkit (NLTK).

You can find the final code for this tutorial in this DigitalOcean repository.