cURL – Split a file and download simultaneously from multiple locations
cURL is a command line tool for transferring files with URL syntax. It supports FTP, FTPS, HTTP, HTTPS, SCP, SFTP, TFTP, TELNET, DICT, LDAP, LDAPS and FILE protocols. With cURL, you can stop a download and then at a later stage, resume from where you left off. This is a great boon for downloading large files.
Here is a tip to split a large file and download it simultaneously from multiple locations. This is especially useful where you are downloading an ISO image of a Linux distribution but the server from which you download the ISO file has throttled the bandwidth.
In this case, you can use curl to download different parts of the ISO image from alternate servers thus circumventing the throttling issue.
As an example, I will show you how to download Ubuntu 8.10 Linux distribution ISO using curl and the above mentioned trick.
Identify the download mirrors from the Ubuntu website. Choose at least 4 different mirrors. I have chosen the following locations :
Where url1, url2, url3 and url4 are variables which are associated with the corresponding full paths of the Ubuntu ISO file.
We are going to split the file
ubuntu-8.10-desktop-i386.iso into 4 parts, with 3 parts of 200 MB and the rest making up the 4th part. So the cURL command(s) will be as follows :
curl --range 0-199999999 -o ubuntu-iso.part1 $url1 & curl --range 200000000-399999999 -o ubuntu-iso.part2 $url2 & curl --range 400000000-599999999 -o ubuntu-iso.part3 $url3 & curl --range 600000000- -o ubuntu-iso.part4 $url4 &
--range option retrieves a byte range in the first three cases 199.99 MB of the file.
-o specifies the output file name and
$url[1..4] are the above mentioned URLs.
For easy execution, I have created a bash script named “
download.sh“, comprising of the above lines and to start the download, I need just run the script.
#FILE NAME : download.sh (Download accelerator) #!/bin/sh url1=http://ftp.daum.net/ubuntu-releases/intrepid/ubuntu-8.10-desktop-i386.iso url2=http://ftp.yz.yamagata-u.ac.jp/pub/linux/ubuntu/releases/intrepid/ubuntu-8.10-desktop-i386.iso url3=http://ie.releases.ubuntu.com/intrepid/ubuntu-8.10-desktop-i386.iso url4=http://releases.ubuntu.com/intrepid/ubuntu-8.10-desktop-i386.iso curl --range 0-199999999 -o ubuntu-iso.part1 $url1 & curl --range 200000000-399999999 -o ubuntu-iso.part2 $url2 & curl --range 400000000-599999999 -o ubuntu-iso.part3 $url3 & curl --range 600000000- -o ubuntu-iso.part4 $url4 &
Run the above script from the command line as follows :
$ chmod ugo+x download.sh $ ./download.sh
Once the download is complete, I will have 4 files named
ubuntu-iso.part[1..4] which I should combine to retrieve the whole ISO image. This is done as follows :
$ cat ubuntu-iso.part? > ubuntu-8.10-desktop-i386.iso
By using cURL this way, it is possible to speed up the download of a file especially if the servers from which the file is downloaded has their speed throttled.