...
Besides for standard sync and backup, use rsync to,
- Copy over network using the faster rsync protocol and measure speed.
- Copy and resume large directory backups.
Rsync is much more reliable and allows for continuation in case of interruption.
The biggest mistake people don't do is to not test their backup by restoring. You'll be surprised to find later that despite no errors, upon restore you've lost some data. This is because failed copies due to special characters is not alway. Rsync has a nice dry-run function so you can test without having to do a full restore.
Backup and Sync
Most used to perform difference only mirror backup of source to destination and shows,
...
Code Block | ||
---|---|---|
| ||
caffeinate rsync [...] # Mac OS X ? systemd-ihibit rsync [...] # Linux |
Backup Interactively
Here is the rsync command for folder to folder example,
Code Block | ||
---|---|---|
| ||
rsync --archive --delete --sparse --verbose --itemize-changes --human-readable --progress /home/tempadmin/source /home/tempadmin/destination |
--dry-run = Use this first to ensure to simulate a run. Especially if you use --delete.
Warning |
---|
Be very careful not to include a trailing slash in source and destination(maybe slash ok in destination but need to test) or you will end up deleting everything in your target. The trailing slash (/) on the source directory modifies the behaviour,
|
--archive = which includes,
...
--delete = delete any files not in the source
--sparse = --verbose =
--itemize-changes =
--human-readable =
--progress =
source = see what's happening
source = what you are syncing
destination = destination may be a directory but the most useful is actually to or another system running the rsync service
Be very careful not to include a trailing slash in source and destination(maybe slash ok here but need to test) or you will end up deleting everything in your target.
Backup to rsync Service
Instead of a directory, the destination may be another system running the rsync service. This method is extremely fast compare to the everyday use AFP or SMB share protocols. However, I believe there are some limitations when it comes to special characters at least on Mac OS X. I'm still figuring this out.
You must have on the other side an RSync service listening and created an account (in this example rSyncUser) with access to the appropriate directory. Many modern NAS have this capability, you just need to turn them on.
Code Block | ||
---|---|---|
| ||
rsync --archive --delete --sparse --compress --verbose --itemize-changes --human-readable --progress /home/tempadmin/source rsync://rsyncUser@destSystem/destination |
--destination =
--compress =
Run rsync Multiple Times
A point of interest is that I find I often need to run rsync more than once, as it often finds discrepancies even right after the first transfer. Keep on running rsync until you stop seeing "files to consider".
Backup via Script
In order to run as a script add the following,
--password-file =
--log-file=
Straight Copy
Simple straight copy and can be used to resume (still need to modify for resume),
Code Block | ||
---|---|---|
| ||
rsync --archive --sparse --compress --verbose --itemize-changes --human-readable --progress /home/tempadmin/tmp/source /home/tempadmin/tmp/destination |
...
More Details Understanding of the Attributes
--archive
Same as -rlptgoD (no -H)
...
This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.
Exclude Unnecessary Mac OS
...
Hidden Files
Within a script,
Code Block | ||
---|---|---|
| ||
# rsync can't handle spaces if just a variable https://stackoverflow.com/questions/19219774/bash-rsync-with-options-as-variable
EXCLUDE_ARRAY=("'$RECYCLE.BIN'" "'$Recycle.Bin'" "'.AppleDB'" "'.AppleDesktop'" "'.AppleDouble'" "'.com.apple.timemachine.supported'" "'.dbfseventsd'" "'.DocumentRevisions-V100*'" "'.DS_Store'" "'.fseventsd'" "'.PKInstallSandboxManager'" "'.Spotlight*'" "'.SymAV*'" "'.symSchedScanLockxz'" "'.TemporaryItems'" "'.Trash*'" "'.vol'" "'.VolumeIcon.icns'" "'Desktop DB'" "'Desktop DF'" "'hiberfil.sys'" "'lost+found'" "'Network Trash Folder'" "'pagefile.sys'" "'Recycled'" "'RECYCLER'" "'System Volume Information'" "'Temporary Items'" "'Thumbs.db'" "'DF'" "'Trash'" "'Folder'" "'Volume'" "'Information'" "'Items'") |
And then add the variable at the beginning of your execution line as follows,
Code Block | ||
---|---|---|
| ||
caffeinate -s rsync "${EXCLUDE_ARRAY[@]/#/--exclude=}" --archive ... |
You can also do this from the command line too, but the syntax will be different... (still to document).
Understanding the rsync Progress
...
Panel |
---|
|
Example output (and I'll add more details here),
Panel |
---|
|
Mac OS X to Errors
Use this only if things don't work on Mac OS X with the error message noted. On certain (I've yet to determine) versions of Mac OS X, restrictions have increased.
Special Character Issues btw UTF-8 Mac and UTF-8
...
Terminal Restriction
Before using rsync you need to lift FDA (full disk access) restrictions in System Preferences > Security & Privacy > Privacy > Full Disk Access and add Terminal otherwise you may see the error messages when trying to rsync.
...
Copy extended attributes and resource forks.
Exclude unnecessary Mac OS X System Files
List from Alan Smith,
Code Block |
---|
--exclude='$RECYCLE.BIN' --exclude='$Recycle.Bin' --exclude='.AppleDB' --exclude='.AppleDesktop' --exclude='.AppleDouble' --exclude='.com.apple.timemachine.supported' --exclude='.dbfseventsd' --exclude='.DocumentRevisions-V100*' --exclude='.DS_Store' --exclude='.fseventsd' --exclude='.PKInstallSandboxManager' --exclude='.Spotlight*' --exclude='.SymAV*' --exclude='.symSchedScanLockxz' --exclude='.TemporaryItems' --exclude='.Trash*' --exclude='.vol' --exclude='.VolumeIcon.icns' --exclude='Desktop DB' --exclude='Desktop DF' --exclude='hiberfil.sys' --exclude='lost+found' --exclude='Network Trash Folder' --exclude='pagefile.sys' --exclude='Recycled' --exclude='RECYCLER' --exclude='System Volume Information' --exclude='Temporary Items' --exclude='Thumbs.db' |
Over SSH Protocol
rsync ....
...
Solution to FUSE or NFS on MAC - https://stackoverflow.com/questions/667992/rsync-error-failed-to-set-times-on-foo-bar-operation-not-permitted/668049#668049
Understanding what the Progress Bar Looks like - https://stackoverflow.com/questions/4493525/what-does-f-mean-in-rsync-logs
Special Character and Platform Difference Issues - https://askubuntu.com/questions/533690/rsync-with-special-character-files-not-working-between-mac-and-linux https://apple.stackexchange.com/questions/148799/rsync-with-linux-server-special-character-problem