How to Download Files with cURL

Client URL, or cURL, is a library and command-line utility for transferring data between systems. It supports many protocols and tends to be installed by default on many Unix-like operating systems. Because of its general availability, it is a great choice for when you need to download a file to your local system, especially in a server environment.

In this tutorial, you’ll use the curl command to download a text file from a web server. You’ll view its contents, save it locally, and tell curl to follow redirects if files have moved.

Downloading files off of the Internet can be dangerous, so be sure you are downloading from reputable sources. In this tutorial you’ll download files from DigitalOcean, and you won’t be executing any files you download.

Step 1 — Fetching remote files

Out of the box, without any command-line arguments, the curl command will fetch a file and display its contents to the standard output.

Let’s give it a try by downloading the robots.txt file from Digitalocean.com:

curl https://www.digitalocean.com/robots.txt

You’ll see the file’s contents displayed on the screen:

Output
User-agent: *
Disallow:

sitemap: https://www.digitalocean.com/sitemap.xml
sitemap: https://www.digitalocean.com/community/main_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/questions_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/users_sitemap.xml.gz

Give curl a URL and it will fetch the resource and display its contents.

Saving Remote Files

Fetching a file and display its contents is all well and good, but what if you want to actually save the file to your system?

To save the remote file to your local system, with the same filename as the server you’re downloading from, add the --remote-name argument, or use the -O option:

curl -O https://www.digitalocean.com/robots.txt

Your file will download:

Output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   286    0   286    0     0   5296      0 --:--:-- --:--:-- --:--:--  5296

Instead of displaying the contents of the file, curl displays a text-based progress meter and saves the file to the same name as the remote file’s name. You can check on things with the cat command:

cat robots.txt

The file contains the same contents you saw previously:

Output
User-agent: *
Disallow:

sitemap: https://www.digitalocean.com/sitemap.xml
sitemap: https://www.digitalocean.com/community/main_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/questions_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/users_sitemap.xml.gz

Now let’s look at specifying a filename for the downloaded file.

Step 2 — Saving Remote Files with a Specific File Name

You may already have a local file with the same name as the file on the remote server.

To avoid overwriting your local file of the same name, use the -o or --output argument, followed by the name of the local file you’d like to save the contents to.

Execute the following command to download the remote robots.txt file to the locally named do-bots.txt file:

curl -o do-bots.txt  https://www.digitalocean.com/robots.txt

Once again you’ll see the progress bar:

Output
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   286    0   286    0     0   6975      0 --:--:-- --:--:-- --:--:--  7150

Now use the cat command to display the contents of do-bots.txt to verify it’s the file you downloaded:

cat do-bots.txt

The contents are the same:

Output
User-agent: *
Disallow:

sitemap: https://www.digitalocean.com/sitemap.xml
sitemap: https://www.digitalocean.com/community/main_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/questions_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/users_sitemap.xml.gz

By default, curl doesn’t follow redirects, so when files move, you might not get what you expect. Let’s look at how to fix that.

Step 3 — Following Redirects

Thus far all of the examples have included fully qualified URLs that include the https:// protocol. If you happened to try to fetch the robots.txt file and only specified www.digitalocean.com, you would not see any output, because DigitalOcean redirects requests from http:// to https://:

You can verify this by using the -I flag, which displays the request headers rather than the contents of the file:

curl -I www.digitalocean.com/robots.txt

The output shows that the URL was redirected. The first line of the output tells you that it was moved, and the Location line tells you where:

Output
HTTP/1.1 301 Moved Permanently
Cache-Control: max-age=3600
Cf-Ray: 65dd51678fd93ff7-YYZ
Cf-Request-Id: 0a9e3134b500003ff72b9d0000000001
Connection: keep-alive
Date: Fri, 11 Jun 2021 19:41:37 GMT
Expires: Fri, 11 Jun 2021 20:41:37 GMT
Location: https://www.digitalocean.com/robots.txt
Server: cloudflare
. . .

You could use curl to make another request manually, or you can use the --location or -L argument which tells curl to redo the request to the new location whenever it encounters a redirect. Give it a try:

curl -L www.digitalocean.com/robots.txt

This time you see the output, as curl followed the redirect:

Output
User-agent: *
Disallow:

sitemap: https://www.digitalocean.com/sitemap.xml
sitemap: https://www.digitalocean.com/community/main_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/questions_sitemap.xml.gz
sitemap: https://www.digitalocean.com/community/users_sitemap.xml.gz

You can combine the -L argument with some of the aforementioned arguments to download the file to your local system:

curl -L -o do-bots.txt www.digitalocean.com/robots.txt

Conclusion

curl lets you quickly download files from a remote system. curl supports many different protocols and can also make more complex web requests, including interacting with remote APIs to send and receive data.

You can learn more by viewing the manual page for curl by running man curl.