Common Usage
...
Introduction
Besides for standard sync and backup, use rsync to,
- Copy over network using the faster rsync protocol and measure speed.
- Copy and resume large directory backups.
Rsync is much more reliable and allows for continuation in case of interruption.
The biggest mistake people don't do is to not test their backup by restoring. You'll be surprised to find later that despite no errors, upon restore you've lost some data. This is because failed copies due to special characters is not alway. Rsync has a nice dry-run function so you can test without having to do a full restore.
Backup and Sync
Most used to perform difference only mirror backup of source to destination and shows,
- Progress of copy
- Speed of copy
- Summary of bytes sent & received, total size and average transfer speed
On a desktop machines that might go to sleep using with a keep alive command,
Code Block | ||
---|---|---|
| ||
caffeinate rsync [...] # Mac OS X
systemd-ihibit rsync [...] # Linux |
Backup Interactively
Here is the rsync command for folder to folder example,
Code Block | ||
---|---|---|
| ||
rsync --archive --delete --sparse --compressverbose --verboseitemize-changes --human-readable --progress /home/tempadmin/tmp/source source /home/tempadmin/destination |
--dry-run = Use this first to ensure to simulate a run. Especially if you use --delete.
Warning |
---|
Be very careful not to include a trailing slash in source and destination(maybe slash ok in destination but need to test) or you will end up deleting everything in your target. The trailing slash (/) on the source directory modifies the behaviour,
|
--archive = which includes,
- --recursive = subdirectories
- --links = copy symbolic links as symbolic links
- --perms = preserve permissions
- --times = preserve times
- --group = preserve group
- --owner = preserve owner
- --devices = preserve device files (super-user only)
- --specials = preserve special files
--delete = delete any files not in the source
--verbose = see what's happening
source = what you are syncing
destination = destination may be a directory or another system running the rsync service
Backup to rsync Service
Instead of a directory, the destination may be another system running the rsync service. This method is extremely fast compare to the everyday use AFP or SMB share protocols. However, I believe there are some limitations when it comes to special characters at least on Mac OS X. I'm still figuring this out.
You must have on the other side an RSync service listening and created an account (in this example rSyncUser) with access to the appropriate directory. Many modern NAS have this capability, you just need to turn them on.
Code Block | ||
---|---|---|
| ||
rsync --archive --delete --sparse --compress --verbose --itemize-changes --human-readable --progress /home/tempadmin/tmp/source rsync://rsyncUser@destSystem/destination |
--destination =
--compress =
Run rsync Multiple Times
A point of interest is that I find I often need to run rsync more than once, as it often finds discrepancies even right after the first transfer. Keep on running rsync until you stop seeing "files to consider".
Backup via Script
In order to run as a script add the following,
--password-file =
--log-file=
More Details Understanding of the Attributes
--archive
Same as -rlptgoD (no -H)
...
Compress files during transfer.
Warning | ||
---|---|---|
| ||
The default list of file extensions that will not be compressed is: |
Panel |
---|
For images, media or any other already compressed files do not compress as you just slow things down. |
--progress
Show progress.
Warning |
---|
Don't use with cron. |
Special Use
--whole-file
Use this for first time sync if you have lots of files.
With this option the incremental rsync algorithm is not used and the the whole file is is sent as-is instead. The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the "disk" is actually a networked filesystem). This is the default when both the source and destination are specified as local paths.
...
This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.
Exclude Unnecessary Mac OS
...
Hidden Files
Within a script,
Code Block | ||
---|---|---|
| ||
# rsync can't handle spaces if just a variable https://stackoverflow.com/questions/19219774/bash-rsync-with-options-as-variable
EXCLUDE_ARRAY=("'$RECYCLE.BIN'" "'$Recycle.Bin'" "'.AppleDB'" "'.AppleDesktop'" "'.AppleDouble'" "'.com.apple.timemachine.supported'" "'.dbfseventsd'" "'.DocumentRevisions-V100*'" "'.DS_Store'" "'.fseventsd'" "'.PKInstallSandboxManager'" "'.Spotlight*'" "'.SymAV*'" "'.symSchedScanLockxz'" "'.TemporaryItems'" "'.Trash*'" "'.vol'" "'.VolumeIcon.icns'" "'Desktop DB'" "'Desktop DF'" "'hiberfil.sys'" "'lost+found'" "'Network Trash Folder'" "'pagefile.sys'" "'Recycled'" "'RECYCLER'" "'System Volume Information'" "'Temporary Items'" "'Thumbs.db'" "'DF'" "'Trash'" "'Folder'" "'Volume'" "'Information'" "'Items'") |
And then add the variable at the beginning of your execution line as follows,
Code Block | ||
---|---|---|
| ||
caffeinate -s rsync "${EXCLUDE_ARRAY[@]/#/--exclude=}" --archive ... |
You can also do this from the command line too, but the syntax will be different... (still to document).
Understanding the rsync Progress
...
Panel |
---|
|
Example output (and I'll add more details here),
Panel |
---|
|
Mac OS X to Errors
Use this only if things don't work on Mac OS X with the error message noted. On certain (I've yet to determine) versions of Mac OS X, restrictions have increased.
Special Character Issues btw UTF-8 Mac and UTF-8
...
Terminal Restriction
Before using rsync you need to lift FDA (full disk access) restrictions in System Preferences > Security & Privacy > Privacy > Full Disk Access and add Terminal otherwise you may see the error messages when trying to rsync.
For me, I ran into this challenge with my Photos Library,
Code Block |
---|
building file list
rsync: opendir "/Users/tin.pham/Pictures/Photos Library.photoslibrary" failed: Operation not permitted (1)
1 file to consider
IO error encountered -- skipping file deletion
sent 102 bytes received 16 bytes 21.45 bytes/sec
total size is 0 speedup is 0.00
rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52.200.2/rsync/main.c(996) [sender=2.6.9] |
After making the System Preferences change, it will ask you to restart your Terminal app for the changes to take effect. Here is how your Privacy will look after Terminal is added,
Permission Problems when Syncing with Your Synology Nas Drive
If you see this error message when syncing with your NAS drive,
Code Block |
---|
rsync: failed to set permissions on "/photos/." (in rsync): Operation not permitted (1) |
...
NFS or FUSE File Restrictions
If copying to or copying from file systems that use mapped drives such as NFS or FUSE, you may run into some trouble seeing set times.
Code Block |
---|
rsync: failed to set times on "." (in rsync): Operation not permitted (1) |
You can suppress this with --omit-dir-times
Extended Attributes
--extended-attributes
Copy extended attributes and resource forks.
Exclude unnecessary Mac OS X System Files
List from Alan Smith,
Code Block |
---|
--exclude='$RECYCLE.BIN' --exclude='$Recycle.Bin' --exclude='.AppleDB' --exclude='.AppleDesktop' --exclude='.AppleDouble' --exclude='.com.apple.timemachine.supported' --exclude='.dbfseventsd' --exclude='.DocumentRevisions-V100*' --exclude='.DS_Store' --exclude='.fseventsd' --exclude='.PKInstallSandboxManager' --exclude='.Spotlight*' --exclude='.SymAV*' --exclude='.symSchedScanLockxz' --exclude='.TemporaryItems' --exclude='.Trash*' --exclude='.vol' --exclude='.VolumeIcon.icns' --exclude='Desktop DB' --exclude='Desktop DF' --exclude='hiberfil.sys' --exclude='lost+found' --exclude='Network Trash Folder' --exclude='pagefile.sys' --exclude='Recycled' --exclude='RECYCLER' --exclude='System Volume Information' --exclude='Temporary Items' --exclude='Thumbs.db' |
Over SSH Protocol
rsync ....
Code Block | ||
---|---|---|
| ||
# Rsync over the Internet rsync --archive --verbose --compress --delete --progress -e "ssh -c arcfour -o Compression=no -x" /source/folder remotebackup@earth.com:/home/user:destination-folder # Rsync over LAN # Same but disable all compression. |
caffeinate -s = prevents OS X from sleeping until command is done
--archive or -a = Archive mode. Performs recursion and preserves almost all attributes of the source files (with -H being a notable omission). Note that it does not preserve hard links, because finding multiply-linked files is expensive. You must separately specify -H.
...
What to copy: -r, --recursive recurse into directories -R, --relative use relative path names --exclude=PATTERN Exclude files matching PATTERN --exclude-from=FILE Read exclude patterns from FILE -I, --ignore-times Don't exclude files that match length and time --size-only only use file size when determining if a file should be transferred --modify-window=NUM Timestamp window (seconds) for file match (default=0) --include=PATTERN Don't exclude files matching PATTERN --include-from=FILE Read include patterns from FILE How to copy it: -n, --dry-run Perform a trial run with no changes made -l, --links Copy symlinks as symlinks -L, --copy-links Transform symlink into referent file/dir --copy-unsafe-links Only "unsafe" symlinks are transformed --safe-links Ignore links outside the destination tree -H, --hard-links Preserve hard links -D, --devices Preserve devices (super-user only) -g, --group Preserve group -o, --owner Preserve owner (super-user only) -p, --perms Preserve permissions -t, --times Preserve times -S, --sparse Handle sparse files efficiently -x, --one-file-system Don't cross filesystem boundaries -B, --block-size=SIZE Force a fixed checksum block-size (default 700) -e, --rsh=COMMAND Specify rsh replacement --rsync-path=PATH Specify path to rsync on the remote machine --numeric-ids Don't map uid/gid values by user/group name --timeout=TIME Set IO timeout in seconds -W, --whole-file Copy whole files, no incremental checks Destination options: -a, --archive Archive mode -b, --backup Make backups (see --suffix & --backup-dir) --backup-dir=DIR Make backups into this directory --suffix=SUFFIX Override backup suffix -z, --compress Compress file data during the transfer -c, --checksum Skip based on checksum, not mod-time & size -C, --cvs-exclude Auto ignore files in the same way CVS does --existing Only update files that already exist --delete Delete files that don't exist on the sending side --delete-excluded also delete excluded files on the receiving side --delete-after Receiver deletes after transfer, not during --force Force deletion of directories even if not empty --ignore-errors Delete even if there are IO errors --max-delete=NUM Don't delete more than NUM files --log-format=FORMAT Log file transfers using specified format --partial Keep partially transferred files --progress Show progress during transfer -P equivalent to --partial --progress --stats Give some file transfer stats -T --temp-dir=DIR Create temporary files in directory DIR --compare-dest=DIR also compare destination files relative to DIR -u, --update update only (don't overwrite newer files) Misc Others: --address=ADDRESS bind to the specified address --blocking-io Use blocking IO for the remote shell --bwlimit=KBPS Limit I/O bandwidth, KBytes per second --config=FILE Specify alternate rsyncd.conf file --daemon Run as a rsync daemon --no-detach Do not detach from the parent --password-file=FILE Get password from FILE --port=PORT Specify alternate rsyncd port number -f, --read-batch=FILE Read batch file -F, --write-batch Write batch file --version Print version number -v, --verbose Increase verbosity -q, --quiet Decrease verbosity -4, --ipv4 Prefer IPv4 -6, --ipv6 Prefer IPv6 -h, --help show this help screen
Warning |
---|
TBC - Roderick |
References
...
For MAC OS X consider - http://osxdaily.com/2009/02/19/command-line-back-ups-in-os-x/
Prevent MAC OX from sleeping - http://www.pcadvisor.co.uk/news/software/3382592/top-20-os-x-command-line-secrets-for-power-users/
Solution to OpenDir Error for Photos on Mac - https://www.reddit.com/r/MacOS/comments/bvo5wt/rsync_error_copying_libraryphotoslibrary/
Solution to FUSE or NFS on MAC - https://stackoverflow.com/questions/667992/rsync-error-failed-to-set-times-on-foo-bar-operation-not-permitted/668049#668049
Understanding what the Progress Bar Looks like - https://stackoverflow.com/questions/4493525/what-does-f-mean-in-rsync-logs
Special Character and Platform Difference Issues - https://askubuntu.com/questions/533690/rsync-with-special-character-files-not-working-between-mac-and-linux https://apple.stackexchange.com/questions/148799/rsync-with-linux-server-special-character-problem