Mac 101: Learn the Power of rsync for Backup, Remote, Archive Systems

You may not have heard of rsync; it’s a file transfer and synchronization program that’s often used to create elaborate and complex backup systems.

Written for Unix operating systems, rsync is included with the Mac and can be accessed directly from Terminal, or used within a number of scripting languages.

The rsync program has a number of features that make it a good candidate for building local, as well as remote, backup, archiving, and synchronization systems. It can also be used for basic file copying, and for maintaining file synchronization between one or more folders, either locally or with a remote system (think cloud-based storage, as an example).

In this Rocket Yard Guide, we’re going to concentrate on using rsync locally. If you wish to use rsync with a remote system, you’ll need to ensure that both the local system and the remote system have rsync installed.

If you’re looking for a copy of rsync to install on a system other than a Mac, or you’re just interested in discovering more about this versatile app, you can check out the rsync website.

Before we get into details about using rsync on the Mac, a note about versions. The version of rsync that’s distributed with the Mac tends to lag behind the current version available on the rsync website. The Mac version has been at 2.6.9 for a number of years, while the current version is at 3.1.3 (as of January, 2018). You should have no problems using the older Mac version with remote platforms that have one of the newer versions installed, but going the other direction could have unexpected results. Always check version compatibility when using rsync with remote systems.

Using Rsync
The Terminal app is used to invoke rsync and its various commands. If you’re new to using the Terminal app, check out the Rocket Yard series Tech 101: Introduction to the Mac’s Terminal App.

Rsync uses a simple structure for issuing commands:

rsync -options theSourceDirectory theDestinationDirectory
While the number of options can get long, the format is always the same; the rsync command followed by any optional switches, then the source directory followed by the destination directory.

Let’s look at a basic rsync command that will copy a directory and all sub-directories it may contain. To tell rsync we want all the files and folders, including everything in subdirectories, we include the -r option. In the Terminal app, enter:

rsync -r /Users/tnelson/Desktop /Volumes/DocsBackup

(Replace tnelson with your user name, and DocsBackup with your desired target for the copy.)
In this example, my messy Desktop folderand its contents will be copied to a USB flash drivenamed DocsBackup. After the command is executed by hitting the return or enter key, the DocsBackup flash drive will have a new folder named Desktop, with all of my Desktop content.

If you want to copy only the contents of the Desktop, and not the parent folder named Desktop, you would add a forward slash after the directory named Desktop, like this:

rsync -r /Users/tnelson/Desktop/ /Volumes/DocsBackup

Terminal Tip:Terminal will figure out the directory pathnames for you. Enter the rsync command and any options at the Terminal prompt, followed by a space, then drag the source directory from a Finder window onto the Terminal window. The source directory pathname will be entered for you. Next, enter a space in Terminal to separate the source and destination directories, then drag the destination directory from a Finder window onto the Terminal window, and the destination pathname will be entered.

Time Stamps
When you copy files and folders to the destination directory, rsync uses the current time and date as the time stamp for the copied files. You can see this by opening a Finder window on the destination, and setting the Finder’s View to List mode. The creation and modification dates for the files will be set to the current time.

This is the default behavior for the rsync program, and having the files reset to the current time can be advantageous in some uses. But if you want to retain the original creation and modification data, you can, through the use of the -t option:

rsync -r -t /Users/tnelson/Desktop /Volumes/DocsBackup

In the above example, we’ve used both the -r and -t options to recursively copy all subdirectories as well as preserve the original time stamps.

rsync Tip:When using multiple option switches, you don’t have to use the dash symbol before each one. Instead, use a single dash, and place the options together. In the above example, rsync -r -t would become rsync -rt

Archive
Another commonly used rsync option is -a. This switch puts rsync into archive mode, which preserves time stamps, performs a recursive copy, keeps all file and directory permissions, preserves owner and group information, and copies any symbolic links.

Archive mode gets a lot of use when you wish to make backups as opposed to just syncing files in a directory. It’s a lot easier to remember the -a option than combining seven or so rsync options together (-rlptgoD) to perform the same task.

If I wanted to back up my Documents directory to the DocsBackup flash drive, I would use the -a (archive) option, as seen in this example:

rsync -a /Users/tnelson/Documents /Volumes/DocsBackup

Synchronize
Since sync is in rsync’s name, it must be a pretty good synchronization tool. The rsync command uses a special algorithm to compare files in the destination and source directories, and only copy the differences to the destination. There are options that allow you to specify how to handle discrepancies between the source and destination:

Delete Files in the Destination
When syncing directories, it’s not uncommon to have files in the destination directory that aren’t present in the source. rsync will normally leave those files alone, but you can also instruct rsync to delete files in the destination directory that don’t appear in the source. This can help ensure that both directories are identical:

rsync -rt –delete /Users/tnelson/Desktop /Volumes/DocsBackup

This new option, –delete, is two dashes plus the word delete. Sometimes, the double dash will be displayed as an em dash, so if you see a long dash instead of two short dashes, be sure to enter two dashes for this option.

The –delete option will remove any file or directory on the destination directory that isn’t present on the source.

A safer way to handle files or folders on the destination that aren’t present on the source is to back them up before they’re removed. rsync can do this for you using the -b and –backup-dir switches:

rsync -rtb –backup-dir=”backup $(date +\%Y-\%m-\%d)” –delete /Users/tnelson/Desktop /Volumes/DocsBackup

In the above command, the -b option has been added to -r and -t, creating the -rtb option. We’ve also used the –backup-dir command to specify the name of the backup folder to use. In this example, the backup folder will be named backup, and have the date, in the form of year, month, day, appended to its name.

You don’t have to append the date to the backup folder name but it helps when the same rsync backup script is used over and over. All that’s actually required for the –backup-dir command is a direct path to be specified, such as:

rsync -rtb –backup-dir=backup –delete /Users/tnelson/Desktop /Volumes/DocsBackup

In the above example, the backup directory named backup is created at the root level of the destination path, so after this command is executed, there will be a new folder named backup at the root of the DocsBackup flash drive.

I also make sure to put the backup option ahead of the delete option to ensure rsync handles the options in the correct order. You don’t want files to be deleted before they’re backed up.

Exclude Files in the Source
Sometimes there may be files or directories in the source that you don’t want to copy to the destination. You could manually move these files before an rsync copy, but an easier way is to use the exclude option:

rsync -a –exclude MyTestFolder /Users/tnelson/Desktop /Volumes/DocsBackup

In the above rsync command, the folder MyTestFolder, which is on the Desktop, will not be copied to the DocsBackup USB flash drive.

Exclude can use a file or folder name, but it also supports the use of regular expressions for pattern matching. Regular expressions and pattern matching would require their own article to do them justice; for now, I suggest opening another Terminal window (Shell, New Window) and entering the following at the Terminal prompt:

man re_syntax

Hit enter or return. This will display information about how to use regular expressions.

Commonly Used Rsync Options
To finish off our introduction to using rsync, let’s take a look at a few common options the command supports.

• -a – Enables archive mode, commonly used to back up a directory and retain all permissions, time stamps, symbolic links, etc.
• -b – Makes backups of any files in the destination directory that aren’t included in the source. Commonly used with –backup-dir and –delete switches.
• –backup-dir= – Allows you to define a name for the backup directory that will be used by the -b option.
• –delete – Removes files present in the destination directory that are absent from the source directory.
• -n – Dry run; no files are moved. Instead, rsync displays what the results would have been. Use -n with the -v option to gather additional information.
• -r – Recursive; forces rsync to transverse all subdirectories contained within the source directory and copy the subdirectories and their content to the destination directory.
• -t – Time stamps; preserves the existing time stamp on files and folders that are being copied.
• -m – Prunes empty directories. Directories that are empty aren’t copied.
• -z – Compress; uses compression during data transfer. Most often used when using rsync with a remote system.
• -v – Verbose; increases the level of information displayed by the rsync command. You can chain multiple v’s together to increase the information presented. Helpful for debugging, but not recommended for general use.

There are many additional rsync options and switches. To see the complete list, open a Terminal window and enter the following at the prompt:

man rsync

Press enter or return.

One of the best ways to understand how rsync works is to give the command a try with some dummy directories. Create a source directory with a few files and subdirectories in it, and try using rsync to copy them to a dummy (test) destination directory.

Be sure and try out the various options and switches and see how they affect the copy process. With a bit of tinkering, you can create customized backup solutions to meet your needs.

Other Topics You Might Like:

Guest Article: How to Bulletproof Your Backups Tech 101: Everything You Need to Know to Verify Time Machine Backups Expand the Possibilities for Your Mac with Data Doubler from OWC Make A Bootable OS X Mavericks USB Install Drive With DiskMaker X