Papandut.Com: like IT, blog IT, share IT, help with IT

Synchronizing Data with Rsync

Rsync is a useful tool that allows files and directory to be copied either to or from a remote host, or locally on the current host. It is a faster and flexible replacement for rcp.
There are two different ways for rsync to contact a remote system:
– using a remote-shell program as the transport (such as ssh or rsh) or
– contacting an rsync daemon directly via TCP.
For remote transfers, a modern rsync uses ssh for its communications, but it may have been configured to use a different remote shell by default, such as rsh or remsh.

Rsync uses the rsync remote-update protocol to greatly speed up file transfers. The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network connection, using an efficient checksum-search algorithm.
The files are compared using their checksums and only the necessary files are transferred. This makes rsync very efficient in mirroring files between source and destination directory within hosts or between hosts. Rsync can also be used to delete files, copy device files, copy symbolic links, preserve ownership and group, copy permissions and synchronize timestamps to achieve an exact copy. Rsync also support both include and exclude pattern so that you can specify exactly which files you want to synchronized.

Please note that rsync ignore any changes made on the system being updated (destination). Any files that were locally modified will be replaced with the copy from the source and any deleted files will be added again if it is still exist in the source. This means that you must make all changes on the master source and never make any changes on the destination system.

To use rsync, you must specify a source and a destination, one of which may be remote.
The syntax is:
rsync [OPTION]... SRC [SRC]... DEST
There are many options to use with rsync.

Here are some samples usages of rsync:
1. Use the following command to: transfer all files from the current directory to the directory /dest.
rsync * /dest

2. Use the following command to: transfer all files matching the pattern *.doc from the current directory to the directory /dest.
rsync *.doc /dest

3. Use the following command to: recursively transfer all content from the directory /src/data1/ on current host to directory /data/temp1. The transfer is in “archive” mode , which ensures that symbolic links, devices, attributes, permissions, ownerships, etc. are preserved in the transfer.
rsync –av /src/data1/ /data/temp1

4. Use the following command to: recursively transfer in archive mode the directory /src1/data1 and /src2/data2 on local host to the directory /dest.
rsync –av /src1/data1 /src2/data2 /dest

5. Use the following command to: recursively transfer in archive mode all content from the directory /src/data1/ on remote host named pc1 to directory /dest/temp1 on local host. Compression will be used to reduce the size of data portions of the transfer.
rsync –avz pc1:/src/data1/ /dest/tmp1

6. Use the following command to: recursively transfer in archive mode all content from the directory /src/data1/ on local host to directory /dest/temp1 on remote host named pc1. This command also tells rsync to delete any files in the target directory that are not present in the source directory. Be carefull with — delete options.
rsync -av --delete /src/data1/ pc1:/dest/temp1

7. Use the following command to: recursively transfer in archive mode all content from the directory /src/data1/ on local host to directory /dest/temp1 on remote host named pc1 but exclude all files with ext. of tmp and temp.
rsync -av –-exclude *.tmp –-exclude *.temp /src/data1/ pc1:/dest/temp1

8. Use the following command to: recursively transfer in archive mode all content from the directory /src/data1/ on local host to directory /dest/temp1 on remote host named pc1 and log on to pc1 using credential of user1.
rsync -av /src/data1/ user1@pc1:/dest/temp1

9. Use the following command to: recursively transfer in archive mode all content from the directory /src/data1/ on local host to directory /dest/temp1 on remote host named pc1, preserve hard links, treat symlink dir as dir and log on to pc1 using credential of user1.
rsync -avHK /src/data1/ user1@pc1:/dest/temp1

Key-based Authentication for SSH (also for SCP and SFTP)

Besides manually providing login credential for authentication, SSH also offer key-based authentication, which uses cryptographic keys to establish a trust relationship between client and server. Key-based authentication can require a password or can operate without a password on the key. The passwordless key-based authentication also known as null-passphrase key is usually used for automating file transfers like backups, password synchronization, and file system synchronization. When using Rsync with SSH for automated backups, you may prefer to enable key-based authentications so backups can be performed via SSH without password prompting.

For security consideration, null-passphrase key-based authentication should not be used when manually providing credentials for an interactive used of SSH session is possible. There is a security risk between the computers that have the key-based trust that would allow an attacker to log in between those computers without a username or password.
By default, key-based authentication is already turn on in the /etc/ssh/sshd_config file. To make sure, open the /etc/ssh/sshd_config file then locate the line beginning with #RSAAuthentication. That line and two lines below it should look like these:

#RSAAuthentication yes
#PubkeyAuthentication yes
#AuthorizedKeysFile .ssh/authorized_keys
If you need to change the configuration file, then you need to restart the sshd:
# /etc/init.d/sshd restart
Stopping sshd: [ OK ]
Starting sshd: [ OK ]
#

When a user (eg. user1) from a client computer request ssh access to a server, sshd in that server will look in /home/user1/.ssh/authorized_keys to see whether this user has an authorized public key that can be trusted as user1’s key.
In order to work with key-based authentication, user1 in the client computer need to generate a public/private key pair then put the public key into the /home/user1/.ssh/authorized_keys file on the server.

The following show step-by-step action to take to perform key-based authentication for a user called “bj”. The user bj has already created on both the client (computer named “fw”) and server (computer named “fledge”) to be used for key-based authentication.

1. While logged in as the bj user on the client computer, generate a key pair using ssh-keygen command. In this sample, I use null-passphrase.
[bj@fw ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/bj/.ssh/id_dsa):
Created directory '/home/bj/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/bj/.ssh/id_dsa.
Your public key has been saved in /home/bj/.ssh/id_dsa.pub.
The key fingerprint is:
e5:8b:34:7f:83:a0:61:02:86:ec:18:93:2c:4b:e4:e6 [email protected]
[bj@fw ~]$ ls -la .ssh/
total 16
drwx------ 2 bj bj 4096 Jan 3 15:17 .
drwx------ 3 bj bj 4096 Jan 3 15:17 ..
-rw------- 1 bj bj 668 Jan 3 15:17 id_dsa
-rw-r--r-- 1 bj bj 608 Jan 3 15:17 id_dsa.pub
The output of the ssh-keygen command are bj’s private and public (.pub) dsa-based keys on his client computer.

2. Transfer the public key to the server. Assume that the server ip address is 1.2.3.1 and I use SCP to transfer the file.
[bj@fw ~]$ scp .ssh/id_dsa.pub [email protected]:id_dsa-bj-at-fw.pub
The authenticity of host '1.2.3.1 (1.2.3.1)' can't be established.
RSA key fingerprint is 8d:35:1e:e4:1a:87:26:9a:b0:96:c3:5d:cd:1b:c1:ed.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '1.2.3.1' (RSA) to the list of known hosts.
[email protected]'s password:
id_dsa.pub 100% 608 0.6KB/s 00:00
[bj@fw ~]$
I was asked whether I’d like to continue connecting to the server since the authenticity can’t be verified. This is normal behavior, because this is the first time I connect to the SSH server, so I type “yes” to continue connecting. After supplying the password for the user bj on the server, the file is transferred.

3. Log on to the server as user bj. Once on the server, create an .ssh directory within bj’s home directory, if one doesn’t already exist, and place the contents of the id_dsa-bj-at-fw.pub file into a file called .ssh/authorized_keys.
[bj@fw ~]$ ssh [email protected]
[email protected]'s password:
[bj@fledge ~]$ mkdir .ssh
[bj@fledge ~]$ cat id_dsa-bj-at-fw.pub >> .ssh/authorized_keys
[bj@fledge ~]$

4. Verify the permission of the .ssh directory and authorized_keys file. Make sure that they can only be accessed by the user who owns them.
[bj@fledge ~]$ chmod 700 .ssh
[bj@fledge ~]$ chmod 700 .ssh/authorized_keys

5. Get user bj logs out from the server and then logs back in to see if the authorized_keys entry worked. If login no longer requires a password, then the key-based ssh authentication is working.
[bj@fledge ~]$ exit
Connection to 1.2.3.1 closed.
[bj@fw ~]$ ssh [email protected]
Last login: Sat Jan 3 12:03:27 2009 from 1.2.3.2
[bj@fledge ~]$

With this in place, user bj can now transfer files between client computer and server using Rsync via SSH, SCP and SFTP without supplying the password. This facility can then be used for automated backups that need no user intervention in supplying the password when performing Rsync to/from remote host. In next article, I will discuss about how to perform automated backup using Rsync.