...

Besides for standard sync and backup, use rsync to,

1. Copy over network using the faster rsync protocol and measure speed.
2. Copy and resume large directory backups.

...

Rsync is much more reliable and allows for continuation in case of interruption.

The biggest mistake people don't do is to not test their backup by restoring. You'll be surprised to find later that despite no errors, upon restore you've lost some data. This is because failed copies due to special characters is not alway. Rsync has a nice dry-run function so you can test without having to do a full restore.

Backup and Sync

Most used to perform difference only mirror backup of source to destination and shows,

...

caffeinate rsync [...] # Mac OS X
? systemd-ihibit rsync [...] # Linux

Backup Interactively

Here is the rsync command for folder to folder example,

rsync --archive --delete --sparse --verbose --itemize-changes --human-readable --progress /home/tempadmin/source /home/tempadmin/destination

--dry-run = Use this first to ensure to simulate a run. Especially if you use --delete.

--archive = which includes,

...

--delete = delete any files not in the source

--sparse = --verbose =  

--itemize-changes = 

--human-readable =

--progress = 

source = see what's happening

source = what you are syncing

destination = destination may be a directory but the most useful is actually to or another system running the rsync service

Be very careful not to include a trailing slash in source and destination(maybe slash ok here but need to test) or you will end up deleting everything in your target.

Backup to rsync Service

Instead of a directory, the destination may be another system running the rsync service. This method is extremely fast compare to the everyday use AFP or SMB share protocols. However, I believe there are some limitations when it comes to special characters at least on Mac OS X. I'm still figuring this out.

You must have on the other side an RSync service listening and created an account (in this example rSyncUser) with access to the appropriate directory. Many modern NAS have this capability, you just need to turn them on.

rsync --archive --delete --sparse --compress --verbose --itemize-changes --human-readable --progress /home/tempadmin/source rsync://rsyncUser@destSystem/destination

--destination = 

--compress = 

Run rsync Multiple Times

A point of interest is that I find I often need to run rsync more than once, as it often finds discrepancies even right after the first transfer. Keep on running rsync until you stop seeing "files to consider".

Backup via Script

In order to run as a script add the following,

--password-file = 

--log-file= 

Straight Copy

Simple straight copy and can be used to resume (still need to modify for resume),

 rsync --archive --sparse --compress --verbose --itemize-changes --human-readable --progress /home/tempadmin/tmp/source /home/tempadmin/tmp/destination

...

More Details Understanding of the Attributes

--archive

Same as -rlptgoD (no -H) 

...

This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.

Exclude Unnecessary Mac OS

...

Hidden Files

Within a script,

# rsync can't handle spaces if just a variable https://stackoverflow.com/questions/19219774/bash-rsync-with-options-as-variable
EXCLUDE_ARRAY=("'$RECYCLE.BIN'" "'$Recycle.Bin'" "'.AppleDB'" "'.AppleDesktop'" "'.AppleDouble'" "'.com.apple.timemachine.supported'" "'.dbfseventsd'" "'.DocumentRevisions-V100*'" "'.DS_Store'" "'.fseventsd'" "'.PKInstallSandboxManager'" "'.Spotlight*'" "'.SymAV*'" "'.symSchedScanLockxz'" "'.TemporaryItems'" "'.Trash*'" "'.vol'" "'.VolumeIcon.icns'" "'Desktop DB'" "'Desktop DF'" "'hiberfil.sys'" "'lost+found'" "'Network Trash Folder'" "'pagefile.sys'" "'Recycled'" "'RECYCLER'" "'System Volume Information'" "'Temporary Items'" "'Thumbs.db'" "'DF'" "'Trash'" "'Folder'"  "'Volume'"  "'Information'" "'Items'")

And then add the variable at the beginning of your execution line as follows,

caffeinate -s rsync "${EXCLUDE_ARRAY[@]/#/--exclude=}" --archive ...

You can also do this from the command line too, but the syntax will be different... (still to document).

Understanding the rsync Progress

...

YXcstpoguax  path/to/file
|||||||||||
||||||||||╰- x: The extended attribute information changed
|||||||||╰-- a: The ACL information changed
||||||||╰--- u: The u slot is reserved for future use
|||||||╰---- g: Group is different
||||||╰----- o: Owner is different
|||||╰------ p: Permission are different
||||╰------- t: Modification time is different
|||╰-------- s: Size is different
||╰--------- c: Different checksum (for regular files), or
||              changed value (for symlinks, devices, and special files)
|╰---------- the file type:
|            f: for a file,
|            d: for a directory,
|            L: for a symlink,
|            D: for a device,
|            S: for a special file (e.g. named sockets and fifos)
╰----------- the type of update being done::
             <: file is being transferred to the remote host (sent)
             >: file is being transferred to the local host (received)
             c: local change/creation for the item, such as:
                - the creation of a directory
                - the changing of a symlink,
                - etc.
             h: the item is a hard link to another item (requires 
                --hard-links).
             .: the item is not being updated (though it might have
                attributes that are being modified)
             *: means that the rest of the itemized-output area contains
                a message (e.g. "deleting")

Example output (and I'll add more details here),

>f+++++++++ some/dir/new-file.txt
.f....og..x some/dir/existing-file-with-changed-owner-and-group.txt
.f........x some/dir/existing-file-with-changed-unnamed-attribute.txt
>f...p....x some/dir/existing-file-with-changed-permissions.txt
>f..t..g..x some/dir/existing-file-with-changed-time-and-group.txt
>f.s......x some/dir/existing-file-with-changed-size.txt
>f.st.....x some/dir/existing-file-with-changed-size-and-time-stamp.txt 
cd+++++++++ some/dir/new-directory/
.d....og... some/dir/existing-directory-with-changed-owner-and-group/
.d..t...... some/dir/existing-directory-with-different-time-stamp/

Mac OS X to Errors

Use this only if things don't work on Mac OS X with the error message noted. On certain (I've yet to determine) versions of Mac OS X, restrictions have increased.

Special Character Issues btw UTF-8 Mac and UTF-8

...

Terminal Restriction

Before using rsync you need to lift FDA (full disk access) restrictions in System Preferences > Security & Privacy > Privacy > Full Disk Access and add Terminal otherwise you may see the error messages when trying to rsync.

For me, I ran into this challenge with my Photos Library, 

building file list
rsync: opendir "/Users/tin.pham/Pictures/Photos Library.photoslibrary" failed: Operation not permitted (1)
1 file to consider
IO error encountered -- skipping file deletion
sent 102 bytes  received 16 bytes  21.45 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-52.200.2/rsync/main.c(996) [sender=2.6.9]

After making the System Preferences change, it will ask you to restart your Terminal app for the changes to take effect. Here is how your Privacy will look after Terminal is added,

Image Added

Permission Problems when Syncing with Your Synology Nas Drive

If you see this error message when syncing with your NAS drive,

rsync: failed to set permissions on "/photos/." (in rsync): Operation not permitted (1)

...

NFS or FUSE File Restrictions

If copying to or copying from file systems that use mapped drives such as NFS or FUSE, you may run into some trouble seeing set times.

rsync: failed to set times on "." (in rsync): Operation not permitted (1)

You can suppress this with --omit-dir-times

Extended Attributes

--extended-attributes

Copy extended attributes and resource forks.

Exclude unnecessary Mac OS X System Files

List from Alan Smith,

--exclude='$RECYCLE.BIN' --exclude='$Recycle.Bin' --exclude='.AppleDB' --exclude='.AppleDesktop' --exclude='.AppleDouble' --exclude='.com.apple.timemachine.supported' --exclude='.dbfseventsd' --exclude='.DocumentRevisions-V100*' --exclude='.DS_Store' --exclude='.fseventsd' --exclude='.PKInstallSandboxManager' --exclude='.Spotlight*' --exclude='.SymAV*' --exclude='.symSchedScanLockxz' --exclude='.TemporaryItems' --exclude='.Trash*' --exclude='.vol' --exclude='.VolumeIcon.icns' --exclude='Desktop DB' --exclude='Desktop DF' --exclude='hiberfil.sys' --exclude='lost+found' --exclude='Network Trash Folder' --exclude='pagefile.sys' --exclude='Recycled' --exclude='RECYCLER' --exclude='System Volume Information' --exclude='Temporary Items' --exclude='Thumbs.db'

Over SSH Protocol

rsync ....

...

What to copy:
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
     --exclude=PATTERN       Exclude files matching PATTERN
     --exclude-from=FILE     Read exclude patterns from FILE
 -I, --ignore-times          Don't exclude files that match length and time
     --size-only             only use file size when determining if a file should be transferred
     --modify-window=NUM     Timestamp window (seconds) for file match (default=0)
     --include=PATTERN       Don't exclude files matching PATTERN
     --include-from=FILE     Read include patterns from FILE

How to copy it:  -n, --dry-run               Perform a trial run with no changes made
 -l, --links                 Copy symlinks as symlinks
 -L, --copy-links            Transform symlink into referent file/dir
     --copy-unsafe-links     Only "unsafe" symlinks are transformed
     --safe-links            Ignore links outside the destination tree
 -H, --hard-links            Preserve hard links
 -D, --devices               Preserve devices (super-user only)
 -g, --group                 Preserve group
 -o, --owner                 Preserve owner (super-user only)
 -p, --perms                 Preserve permissions
 -t, --times                 Preserve times
 -S, --sparse                Handle sparse files efficiently
 -x, --one-file-system       Don't cross filesystem boundaries
 -B, --block-size=SIZE       Force a fixed checksum block-size (default 700)
 -e, --rsh=COMMAND           Specify rsh replacement
     --rsync-path=PATH       Specify path to rsync on the remote machine
     --numeric-ids           Don't map uid/gid values by user/group name
     --timeout=TIME          Set IO timeout in seconds
 -W, --whole-file            Copy whole files, no incremental checks

Destination options:  -a, --archive               Archive mode
 -b, --backup                Make backups (see --suffix & --backup-dir)
     --backup-dir=DIR        Make backups into this directory
     --suffix=SUFFIX         Override backup suffix
 -z, --compress              Compress file data during the transfer
 -c, --checksum              Skip based on checksum, not mod-time & size
 -C, --cvs-exclude           Auto ignore files in the same way CVS does
     --existing              Only update files that already exist
     --delete                Delete files that don't exist on the sending side
     --delete-excluded       also delete excluded files on the receiving side
     --delete-after          Receiver deletes after transfer, not during
     --force                 Force deletion of directories even if not empty
     --ignore-errors         Delete even if there are IO errors
     --max-delete=NUM        Don't delete more than NUM files
     --log-format=FORMAT     Log file transfers using specified format
     --partial               Keep partially transferred files
     --progress              Show progress during transfer
 -P                          equivalent to --partial --progress
     --stats                 Give some file transfer stats
 -T  --temp-dir=DIR          Create temporary files in directory DIR
     --compare-dest=DIR      also compare destination files relative to DIR
 -u, --update                update only (don't overwrite newer files)

Misc Others:      --address=ADDRESS       bind to the specified address
     --blocking-io           Use blocking IO for the remote shell
     --bwlimit=KBPS          Limit I/O bandwidth, KBytes per second
     --config=FILE           Specify alternate rsyncd.conf file
     --daemon                Run as a rsync daemon
     --no-detach             Do not detach from the parent
     --password-file=FILE    Get password from FILE
     --port=PORT             Specify alternate rsyncd port number
 -f, --read-batch=FILE       Read batch file
 -F, --write-batch           Write batch file
     --version               Print version number
 -v, --verbose               Increase verbosity
 -q, --quiet                 Decrease verbosity
 -4, --ipv4                  Prefer IPv4
 -6, --ipv6                  Prefer IPv6
 -h, --help                  show this help screen

References

...

Prevent MAC OX from sleeping - http://www.pcadvisor.co.uk/news/software/3382592/top-20-os-x-command-line-secrets-for-power-users/

Solution to OpenDir Error for Photos on Mac - https://www.reddit.com/r/MacOS/comments/bvo5wt/rsync_error_copying_libraryphotoslibrary/

Solution to FUSE or NFS on MAC - https://stackoverflow.com/questions/667992/rsync-error-failed-to-set-times-on-foo-bar-operation-not-permitted/668049#668049

Understanding what the Progress Bar Looks like - https://stackoverflow.com/questions/4493525/what-does-f-mean-in-rsync-logs

Special Character and Platform Difference Issues - https://askubuntu.com/questions/533690/rsync-with-special-character-files-not-working-between-mac-and-linux https://apple.stackexchange.com/questions/148799/rsync-with-linux-server-special-character-problem