Segmented Download

Segmented DownLoads

Segmented Download Information

Take a video file sisters.avi, 630MB in size. The normal means of downloading this file is a single long read operation. If at any stage the download fails the only course available is to download the file again.

It is now possible to download the file in segments , default 10MB. A file is now downloaded in multiple chunks and then stitched together. So for the file above 63 separate segments of 10MB will be downloaded and pieced together. If as an example the 34 segment fails then only this segment is redownloaded.

Pros

The main advantage is given above, that only a segment needs redownloading after a failure
Considering the example above, the average failure in the normal download will waste 50%, 340MB whereas the segmented download will waste 10MB charged at $0.18 GB.

Cons

Taking the above as an example 63 separate downloads will produce some overhead which must be paid for as 63 * overhead per download * $0.18 / 1GB. I do not know the overhead but suspect it is approximate 100 bytes, so for safety call it 1KB the above is neglible

Also each 10,000 requests cost $0.01 and I have gone from 1 request to 63 above, so there is an extra payment of 63 * $0.01 / 10,000 again neglible. Of course someone maybe downloading millions of files per day and these calculation may have relevance.

Segmented Download Instructions

The normal / default download method is to read in one operation.
To download in segments, Options / Options tab Transfer Options and the click Segmented Downloads. On the same page the Segmented Download size can also be changed from the default of 10MB if another value suits you.

The actual Sissle download procedure in choosing and initiating segmented downloads is exactly the same as for normal downloads.

Information: Recovery From Failures in Segmented Downloads

Sissle creates in the application folder a temporary work folder called sisters.avi_____2007_07_04_11_54_03 , this is obviously the file name and the date the file was downloaded. The folder contains the file being downloaded and a separate file splits.txt which contains status information of the download, for instance how much has been downloaded and a lot of far more complex data concerning MD5 hash, gzip, ... processing.

There are 2 types of recovery from a failure in a segmented download.
The first is automatic and will immediately retry the failed segment and on success continue.

There is a second failure procedure which follows after the first recovery above also fails. When the program enters this state, the recovery procedure can be entered at any time, and is initiated by the user. For instance if there is some failure locally in your ISP network you can wait several hours or days even before you continue the download.

Instructions: Recovery From Failures in Segmented Downloads

- Initiate by Options / Retry Download
- This will present a list of failed downloads and their status, select a file to recover, click ok,
- A more detailed box will show more  information, typically how much has been received and the files total size. If you are happy with this information, click Recover and the file will be downloaded from the point of failure.

Advice

The segment size is the amount requested and downloaded in a single transaction with S3, if there is any failure then the download is restarted by requesting the failed section. There is a judgment on the value assigned to segment size, for instance if your files are typically 1.0 GB then a segment size of 20-100MB can be justified. There is a time cost (and a small bandwidth cost) in downloading a file in say 1000 sections as against in 1 lump. If the segment size is greater than what you download in about 1 minute then the extra time is hardly noticed. Each user should conduct his own experiments here to find a value or set of values that apply to his needs.

This mode, like the original mode, now also supports MD5 checking, encryption and zipping, under the same conditions. e.

This is only for large files, also a small segment size of say 1KB which I used for testing has a vast impact on download speed, default is 10MB.
This mode is not default.

Amazon S3 unfortunately only supports this on download and there is no upload version.