Syncany

Distributed downloads

Asked by Jon Gjengset

Is there any chance we might see Syncany supporting multiple storage repositories for one folder (e.g. my "Documents" folder is synced both to two of my SSH servers and an FTP server)?
This would also help download speeds a lot since during an initial sync of a folder, Syncany could download different chunks from different sources concurrently!

Question information

Language:
English Edit question
Status:
Answered
For:
Syncany Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Philipp C. Heckel (binwiederhier) said : 		#1

Hi Jon,

this would of course be a nice thing to have. Especially in terms of high
availabilty. For the first stable release, I don't see this happening, but
in later releases maybe.

If neither of the storages fail, this is not difficult to implement -- a
"Multiple Storages" plugin could easily solve this. It gets tricky as soon
as one of the storages fails, or is unavailable for a couple of minutes.
Because then the two repositories have different states ...

Does that answer your question?

Cheers,
Philipp

Revision history for this message
Jon Gjengset (jonhoo) said : 		#2

It does.
I really hope to see it in later releases. It would provide a lot of advantages to Syncany over its competitors as well!
With regards to failed nodes, you could just treat that as non-updated node, and push a diff of a node that was online the entire period onto the outdated node?

Jon

Revision history for this message
Philipp C. Heckel (binwiederhier) said : 		#3

Replication is never easy, especially if the nodes cannot communicate directly. After all, these nodes are dumb storage servers, which cannot do any processing or transferring. Updates have to be pushed by the client(s). The more clients participate, the more complicated it gets...

But I agree, it would be a killer feature :-D

Revision history for this message
Berthold von Treum (ikillinit) said : 		#4

Couldn't the chunks be shared among different storage repositories, with a certain amount of redundancy ? In this manner the files could be retrieved in the case of a definitive storage loss.

Revision history for this message
Philipp C. Heckel (binwiederhier) said : 		#5

There are two options: (a) Expand the storage by concatenating multiple storages (2GB-FTP + 2GB-IMAP = 4GB), or (b) using them to create reduncancy (2GB-FTP + 2GB-IMAP >= 2GB) ~ mirroring

Both are easy to realize if all storages are working and reachable, but (as I said above) it gets more complicated when one storage fails for 10minutes or even longer. Then the clients have to detect this failure and not use the storage until they are in-sync again. This is not trivial.

Can you help with this problem?

Provide an answer of your own, or ask Jon Gjengset for more information if necessary.

To post a message you must log in.