What's the use of segmented downloading?

Segmented downloading allows you to download different parts of a file from different people at the same time. This can let you get the file faster, when more than one person has it. It also means you can sometimes get a complete file even when no one has the entire thing. It is only necessary that each piece is available.

Segmented uploading allows you to serve more people in a given amount of time, though each one will get smaller pieces. The idea is that something is better than nothing, so it gives something to as many people as possible. It's similar to the difference between an escalator and an elevator; slower, but less wait to get on.

Thx for the explanation but I'm a step ahead. If all upload slots are in use, which is normally the case, what does one gain by downloading chunks? Average download speed of file will be just a bit lower compared to downloading the entire file at once (due to numerous reconnections). And in practice segmented uploading neither allows me to serve more people because slots are not freed completely between chunks, users keep downloading chunks until complete file is finished.

If you download a file from only one person, segmented downloading doesn't help. It makes things a little bit slower. However, if more than one person has the file, it lets you download from multiple people at once, and get it faster. So, the worst case is not quite as good as the non-segmented version, but the best case can be many many times faster. It's usually a net gain.

If the number of people waiting for a file from you is as small as your number of upload slots, segmented uploading doesn't allow you to serve more people in a given amount of time. However, it does still give them the benefits I mentioned above. And, if the number of people wanting your content is larger than your number of slots, it allows you to ration your limited slots to more people, at a cost of slower speed per person. It goes slower, but starts sooner.

Would you rather download at 100KiB/s from one person, or from 20 people at 30KiB/s each (600KiB/s total)? And, would you rather wait in line for an hour before a download starts, or start immediately when you request a file?

I can't say whether this segmented transfer concept helps much in DC++, but it works very well in bittorrent. It's very effective when lots of people are interested in the same file(s), and it doesn't hurt much when the population is small.

Hi Scott, you're assuming that enough clients have free upload slots and unused upload bandwidth. Usually none of both is the case.

Until segmented down- and uploading was introduced it was considered decent to adapt the number of upload slots to the amount of uploadbandwidth to garantee a minimum upload speed (for example 10 kBytes/sec/slot). As by far most internet connection are asymetric it is very probable the upload slots are all in use. Now, to get more out of the segmented downloading one should increase the number of slots (but total upload bandwidth will be still the same). This will cause more overhead so less efficient transfers. eMule is an extreme example of this kind of mechanism. My experience is I get files much faster with DC and Torrent than with eMule, even having 10 times more sources to download from.

> you're assuming that enough clients have free upload slots and unused upload bandwidth.

No, not really. The assumption (and goal, actually) is that everyone's upload capacity is filled all the time. Given that, the system tries to get new downloads started as soon as possible. Then, when a client has a piece of a file, they can start uploading too, and increase the total capacity of the network. If you have to wait in line for an hour to start downloading, that's an hour you can't spend uploading the file you want.

> to get more out of the segmented downloading one should increase the number of slots

No, it's still a good idea to keep the max simultaneous uploads at a reasonable number. It should be low enough to give each person a useful amount of data, but also high enough that you can keep your connection full. Every time an upload finishes, it must go out and find another upload to start. Other uploads should be happening at the same time, to avoid wasted capacity.

> eMule is an extreme example...
> My experience is I get files much faster with DC and Torrent than with eMule,
> even having 10 times more sources to download from.

The speed difference between emule and bittorrent is probably caused by a few things:

  - eMule users tend to share hundreds or thousands of files at once, which only allows a small amount of bandwidth per file. BitTorrent users tend to share a small number of torrents, and can devote more bandwidth to each.
  - Each eMule file tends to be shared by a relatively small number of people, and frequently just one. BitTorrent often has thousands, or even tens of thousands, of people in the same torrent pool. (this varies, of course)
  - eMule's transfer algorithms just don't work as well.

In any case, your original question was "What's the use of segmented downloading?". And here is a summary of its purpose:

  - Allows multiple simultaneous downloads of the same file, which increases overall download speed and reduces the effects of latency and waiting in line.
  - Allows building a complete file out of several incomplete files.
  - Allows clients to upload before they're done downloading, which increases the network capacity sooner and reduces the time needed to deliver a file to everyone.

The discussion of how and whether it works would be more appropriate in a forum.

Does this answer your original question?

You're right about the forum but I don't know of any appropriate one, sorry for this.

Yes, my original question is answered, kind of. I thought I was missing something but as it seems I didn't. In your last post you wrote the crucial issue: "keep the max simultaneous uploads at a reasonable number". For most users "reasonable" means increasing number of upload slots which is a "change of culture" or a slightly new way of doing. Thanks for the short discussion.

One of the largest advantages for me happens when there are two people, one slow with open slots, and one fast without free slots. With chunk downloading, I can start pulling off the slow one and wait for a slot on the fast one at the same time. Without it, I'd have to either live with getting it slowly or disconnect from the slow person and hope I get a slot from the fast person soon enough that I don't regret disconnecting.

Scott: note that sharing of partially completed files has not been implemented yet :) It probably won't be added to NMDC either way, since it will probably require a protocol change (and doing those in NMDC is a PITA).