performance issues and rsync --link-dest option

Asked by paulatz

Dear developers,
I used backintime for a short while, I loved it's easy and effective interface but was quite disappointed with its performance when backing up a directory of many thousand of files to a network drive (either cifs or sshfs) when only a few files are modified. I.e. it takes so long to recreate the directory structure and make the hard links that even if only a few files have changed the entire process takes one hour.

I even rewrote the entire make directories and hard links entirely in python to see if it could be improved, taking care to avoid every unnecessary input/output but with little improvement (I can send yo uthe code if you wish).

On the other hand, I found out that the --link-dest=/last/backup/dir option of rsync does exactly the same job as BiT mkdir+ln+rsync but is several orders of magnitude faster. It can also work directly over ssh, removing the need of cifs or sshfs which contributes quite a bit to the performance boost. So now I don my backups by hand with rsync, which is ok, but I miss the practicity of BiT.

I don't know if you were aware of this option, I would suggest you rely directly on rsync if possible provided how smoothly it works.

bests

Question information

Language:
English Edit question
Status:
Solved
For:
Back In Time Edit question
Assignee:
No assignee Edit question
Solved by:
paulatz
Solved:
Last query:
Last reply:
Revision history for this message
paulatz (justforspam) said :
#1

P.S. I forgot to post the rsync command I use to backup, which I found to be quite reliable:

/usr/bin/rsync -Hzlavub --exclude-from=./RSYNC --link-dest=../2011-09-05/ ~paulatto/./ paulatto@wdwe:/shares/Backup_paulax/$(date +%Y-%m-%d)

Revision history for this message
Dan (danleweb) said :
#2

Hi,

I'm aware of this option but there is a problem with it: the user/group & rights must be the same.
- BIT make snapshots read-only so the rights change and rsync will make a full copy
- if the file-system don't support linux rights (like NTFS) rscyn will make a full copy too

The solution can be:
- an option to make (or not) snapshots read-only
- if snapshots are not "read-only" then detect the file-system (or just another option) and use --link-dest (if possible)

I like the idea so I think I'll try it.
Please open a bug for it.

Regards,
Dan

Revision history for this message
paulatz (justforspam) said :
#3

I found it to be already reported here: https://bugs.launchpad.net/backintime/+filebug in 2009

Revision history for this message
paulatz (justforspam) said :
#4
Revision history for this message
Martin Fisher (yusuf-martin) said :
#5

I would just like to ask whether there have been any further thoughts on this subject (when I back up to my home network drive a backup that involves only a few changed files takes about 3-4 hours. I see this matter has also been touched upon at https://answers.launchpad.net/backintime/+question/93232)?

I would of course love to be able to backup much more quickly with BiT.

With thanks, Martin