How To Get Started With the Requests Library in Python

Introduction

In many web apps, it’s normal to connect to various third-party services by using APIs. When you use these APIs you can get access to data like weather information, sports scores, movie listings, tweets, search engine results, and pictures. You can also use APIs to add functionality to your app. Examples of these are payments, scheduling, emails, translations, maps, and file transfers. If you were to create any of those on your own it would take a lot of time, but with APIs, it can take only minutes to connect to one and access its features and data.

In this article, you will learn about the Python Requests library, which allows you to send HTTP requests in Python.

And since using an API is sending HTTP requests and receiving responses, Requests allows you to use APIs in Python. We’ll demonstrate the use of a language translation API here so you can see an example of how it works.

Prerequisites

To complete this tutorial, you will need:

• This project will require Python installed in a local environment.

Understanding HTTP Requests

HTTP requests are how the web works. Every time you navigate to a web page, your browser makes multiple requests to the web page’s server. The server then responds with all the data necessary to render the page, and your browser then actually renders the page so you can see it.

The generic process is this:

• A client (like a browser or Python script using Requests) will send some data to a URL.
• Then, the server located at the URL will read the data, decide what to do with it, and return a response to the client.
• Finally, the client can decide what to do with the data in the response.

Part of the data the client sends in a request is the request method. Some common request methods are GET, POST, and PUT. GET requests are normally for reading data only without making a change to something, while POST and PUT requests generally are for modifying data on the server. So for example, the Stripe API allows you to use POST requests to create a new charge so a user can purchase something from your app.

When sending a request from a Python script or inside a web app, you, the developer, get to decide what gets sent in each request and what to do with the response. So let’s explore that by first sending a request to DigitalOcean.com and then by using a language translation API.

Step 1 — Installing Python Requests

It is a good idea to create a virtual environment first if you don’t already have one.

Then, you will need to install the library. Let’s install requests using pip.

pip install requests

At this point, your project is ready to use Requests.

Step 2 — Making Your First Request

To start, let’s use Requests for requesting the DigitalOcean site.

Create a file called script.py and add the following code to it. In this article, we won’t have much code to work with, so when something changes you can just update the existing code instead of adding new lines.

import requests

res = requests.get('https://www.digitalocean.com/')

print(res)

So all this code is doing is sending a GET request to DigitalOcean. This is the same type of request your browser sent to view this page, but the only difference is that Requests can’t actually render the HTML, so instead you will just get the raw HTML and the other response information.

You’re using the .get() function here, but Requests allows you to use other functions like .post() and .put() to send those requests as well.

You can run it by executing the script.py file.

python script.py

And here’s what you get in return:

Output
<Response [200]>

Next, let’s investigate what a 200 status code means.

Step 3 — Understanding Status Codes

The first thing you can do is check the status code. HTTP codes range from 1XX to 5XX. Common status codes that you have probably seen are 200, 404, and 500.

Here’s a quick overview of what each status code means:

• 1XX - Information
• 2XX - Success
• 3XX - Redirect
• 4XX - Client Error (you made an error)
• 5XX - Server Error (they made an error)

Generally, what you’re looking for when you perform your own requests are status codes in the 200s.

Requests recognizes that 4XX and 5XX status codes are errors, so if those status codes get returned, the response object from the request evaluates to False.

You can test if a request responded successfully by checking the response for truth. For example:

if res:
    print('Response OK')
else:
    print('Response Failed')

And here’s what you get in return:

Output
<Response [200]>
Response OK

The message "Response Failed" will only appear if a 400 or 500 status code returns. Try changing the URL to some nonsense to see the response fail with a 404.

You can take a look at the status code directly by adding:

print(res.status_code)

This will show you the status code directly so you can check the number yourself.

Output
<Response [404]>
Response Failed
404

This displays the response, the script message, and the response status code.

Step 4 — Understanding Headers

Another thing you can get from the response is the headers. You can take a look at them by using the headers dictionary on the response object.

print(res.headers)

This produces the following output:

Output
{'Date': 'date', 'Content-Type': 'text/html', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Set-Cookie': '__cfduid=id; expires=date; path=/; domain=.digitalocean.com; HttpOnly; SameSite=Lax', 'Last-Modified': 'date', 'Vary': 'Accept-Encoding', 'ETag': 'etag', 'Expires': 'date', 'Cache-Control': 'max-age=time, public', 'Content-Encoding': 'gzip', 'CF-Cache-Status': 'DYNAMIC', 'cf-request-id': 'id', 'Expect-CT': 'max-age=time, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 'Server': 'cloudflare', 'CF-RAY': 'ray'}

Headers are sent along with the request and returned in the response. Headers are used so both the client and the server know how to interpret the data that is being sent and received in the response.

You can see the various headers that are returned. A lot of times you won’t need to use the header information directly, but it’s there if you need it.

The content type is usually the one you may need because it reveals the format of the data (HTML, JSON, PDF, text, etc.). But the content type is normally handled by Requests so you can access the data that gets returned.

Step 5 — Understanding Response Text

And finally, if you take a look at res.text (this works for textual data, like the HTML page you are currently viewing) you can see all the HTML needed to build the home page of DigitalOcean. It won’t be rendered, but you see that it looks like it belongs to DigitalOcean.

print(res.text)

This produces the following output:

Output
<!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="x-ua-compatible" content="ie=edge"/><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"/>

[...]

<title data-react-helmet="true">DigitalOcean – The developer cloud</title><meta data-react-helmet="true" name="description" content="Helping millions of developers easily build, test, manage, and scale applications of any size – faster than ever before."/>

[...]

If you saved this to a file and opened it, you would see something that resembled the DigitalOcean site. In a real situation, multiple requests are made for a single web page to load things like images, scripts, and stylesheets, so if you save only the HTML to a file, it won’t look anything like what the DigitalOcean site looks like in your browser because only a single request was performed to get the HTML data.

Step 6 — Using the Translate API

So now let’s move on to something more interesting. You’ll use the Yandex Translate API to perform a request to translate some text to a different language.

To use the API, first, you need to sign up. After you sign up, go to the Translate API and create an API key. Once you have the API key, add it to your file as a constant. Here’s the link where you can do all those things: https://tech.yandex.com/translate/

API_KEY = 'your yandex api key'

The reason why you need an API key is so Yandex can authenticate us every time you want to use their API. The API key is a lightweight form of authentication because it’s added to the end of the request URL when being sent.

To know which URL you need to send to use the API, you can look at the documentation for Yandex.

If you look there, you’ll see all the information needed to use their Translate API to translate text.

https://translate.yandex.net/api/v1.5/tr.json/translate
 ? key=<API key>
 & text=<text to translate>
 & lang=<translation direction>
 & [format=<text format>]
 & [options=<translation options>]
 & [callback=<name of the callback function>]

When you see a URL with ampersands (&), question marks (?), and equals signs (=), you can be sure that the URL is for GET requests. Those symbols specify the parameters that go along with the URL.

Normally things in square brackets ([]) will be optional. In this case, format, options, and callback are optional. And key, text, and lang are required for the request.

So let’s add some code to send to that URL. You can replace the first request you created with this:

url = 'https://translate.yandex.net/api/v1.5/tr.json/translate'
res = requests.get(url)

There are two ways you can add the parameters. You can either append it to the end of the URL directly, or you can have Requests do it for you. To accomplish the latter, you can create a dictionary for our parameters. The three items you need are the key, the text, and the language. Let’s create the dictionary using the API key, 'Hello' for the text, and 'en-es' as the lang, which means you want to translate from English to Spanish.

If you need to know any other language codes, you can look at this list of ISO 639-1 codes. Focus on the 639-1 column.

You can create a params dictionary by using the dict() function and passing in the keys and values you want in your dictionary.

params = dict(key=API_KEY, text='Hello', lang='en-es')

Now we take the parameters dictionary and pass it to the .get() function.

res = requests.get(url, params=params)

When you pass the parameters this way, Requests will go ahead and add the parameters to the URL for you.

Now, let’s add a print statement for the response text and view what gets returned in the response.

print(res.text)

This produces the following output:

Output
{"code': 200, "lang": "en-es", "text": ["Hola"]}

You will see three things. You will see the status code, which is exactly the same status code of the response itself, you will see the language that you specified, and you will see the translated text inside of the list. So you should see 'Hola' for the translated text.

Try again with en-fr as the language code, and you should see 'Bonjour' in the response now.

params = dict(key=API_KEY, text='Hello', lang='en-fr')

This produces the following output:

Output
{"code": 200, "lang": "en-fr", "text": ["Bonjour"]}

Let’s take a look at the headers for this particular response.

print(res.headers)

This produces the following output:

Output
{'Server': 'nginx/1.6.2', 'Date': 'Sat, 20 Apr 2019 08:57:01 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Content-Length": '46', 'Connection': 'keep-alive', 'Keep-Alive': 'timeout=120', 'X-Content-Type-Options': 'nosniff', 'Cache-Control': 'no-store'}

The headers should be different because you’re communicating with a different server, but in this case the content type is application/json instead of text/html. What this means that the data can be interpreted as JSON.

When application/json is the content type of the response, you are able to have Requests convert the response to a dictionary and list so you can access the data easier.

To have the data parsed as JSON, you use the .json() method on the response object.

If you print it, you’ll see that the data looks the same, but the format is slightly different.

json = res.json()
print(json)

This produces the following output:

Output
{'code': 200, 'lang': 'en-fr', 'text': ['Bonjour']}

The reason why it’s different is that it’s no longer plaintext that you get from res.text. This time it’s a printed version of a dictionary.

Let’s say you want to access the text. Since this is now a dictionary, you can use the text key.

print(json['text'])

This produces the following output:

Output
['Bonjour']

And now you only see the data for that one key. In this case, you are looking at a list of one item, so if you wanted to get that text in the list directly, you could access it by the index.

print(json['text'][0])

This produces the following output:

Output
Bonjour

And now the only thing you see is the translated word.

If you change things in your parameters, you’ll get different results. Let’s change the text to be translated from Hello to Goodbye, change the target language back to Spanish, and send the request again.

params = dict(key=API_KEY, text='Goodbye', lang='en-es')

This produces the following output:

Output
Adiós

Try translating longer text in different languages and see what responses the API gives you.

Step 7 — Handling Translate API Error Cases

Finally, we’ll take a look at an error case. Everything doesn’t always work, so you need to know when that happens.

Try changing your API key by removing one character. When you do this your API key will no longer be valid. Then try sending a request.

If you take a look at the status code, this is what you get:

print(res.status_code)

This produces the following output:

Output
403

When you are using the API, you’ll want to check if things are successful or not so you can handle the error cases according to the needs of your app.

Conclusion

In this article, you learned about the Python Requests library, which allows you to send HTTP requests in Python.

Here’s what you learned:

• How HTTP requests work
• The various status codes possible in a response
• How to send requests and receive responses using the Python Requests library
• How to use a language translation API to translate text
• How to convert application/json content responses to dictionaries

If you’d like to learn more about Python, check out our React topic page for exercises and programming projects.