cURL – Split a file and download simultaneously from multiple locations

cURL is a command line tool for transferring files with URL syntax. It supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS and FILE protocols. With cURL, you can stop a download and then at a later stage, resume from where you left off. This is a great boon for downloading large files.

Here is a tip to split a large file and download it simultaneously from multiple locations. This is especially useful where you are downloading an ISO image of a Linux distribution but the server from which you download the ISO file has throttled the bandwidth.

In this case, you can use curl to download different parts of the ISO image from alternate servers thus circumventing the throttling issue.

As an example, I will show you how to download Ubuntu 8.10 Linux distribution ISO using curl and the above mentioned trick.

Step One

Identify the download mirrors from the Ubuntu website. Choose at least 4 different mirrors. I have chosen the following locations :

url1=http://ftp.daum.net/ubuntu-releases/intrepid/ubuntu-8.10-desktop-i386.iso
url2=http://ftp.yz.yamagata-u.ac.jp/pub/linux/ubuntu/releases/intrepid/ubuntu-8.10-desktop-i386.iso
url3=http://ie.releases.ubuntu.com/intrepid/ubuntu-8.10-desktop-i386.iso
url4=http://releases.ubuntu.com/intrepid/ubuntu-8.10-desktop-i386.iso

Where url1, url2, url3 and url4 are variables which are associated with the corresponding full paths of the Ubuntu ISO file.

Step Two

We are going to split the file ubuntu-8.10-desktop-i386.iso into 4 parts, with 3 parts of 200 MB and the rest making up the 4th part. So the cURL command(s) will be as follows :

curl --range 0-199999999 -o ubuntu-iso.part1 $url1 &
curl --range 200000000-399999999 -o ubuntu-iso.part2 $url2 &
curl --range 400000000-599999999 -o ubuntu-iso.part3 $url3 &
curl --range 600000000- -o ubuntu-iso.part4 $url4 &

--range option retrieves a byte range in the first three cases 199.99 MB of the file. -o specifies the output file name and $url[1..4] are the above mentioned URLs.

For easy execution, I have created a bash script named “download.sh“, comprising of the above lines and to start the download, I need just run the script.

#FILE NAME : download.sh (Download accelerator)
#!/bin/sh

url1=http://ftp.daum.net/ubuntu-releases/intrepid/ubuntu-8.10-desktop-i386.iso
url2=http://ftp.yz.yamagata-u.ac.jp/pub/linux/ubuntu/releases/intrepid/ubuntu-8.10-desktop-i386.iso
url3=http://ie.releases.ubuntu.com/intrepid/ubuntu-8.10-desktop-i386.iso
url4=http://releases.ubuntu.com/intrepid/ubuntu-8.10-desktop-i386.iso

curl --range 0-199999999 -o ubuntu-iso.part1 $url1 &
curl --range 200000000-399999999 -o ubuntu-iso.part2 $url2 &
curl --range 400000000-599999999 -o ubuntu-iso.part3 $url3 &
curl --range 600000000- -o ubuntu-iso.part4 $url4 &

Run the above script from the command line as follows :

$ chmod ugo+x download.sh
$ ./download.sh

Step Three

Once the download is complete, I will have 4 files named ubuntu-iso.part[1..4] which I should combine to retrieve the whole ISO image. This is done as follows :

$ cat ubuntu-iso.part? > ubuntu-8.10-desktop-i386.iso

End Note

By using cURL this way, it is possible to speed up the download of a file especially if the servers from which the file is downloaded has their speed throttled.

Related Posts

• Run Nautilus file manager sans GNOME desktop
• Use wget tool to download entire websites
• Convert a USB Flash IMG File to VDI for use in VirtualBox
• Ubuntu Manpage Repository
• Subdownloader – Download Movie Subtitles from Opensubtitles