Try To Use rsync To Complete scp Failed Job

Here is a little trick/tip for nerds who manage websites with a server which allows tools such as SSH and Rsync.  Basically, let’s say you were doing scp command such as …

[scp -r -Cpv* ~/Download/backup/]

but your Internet connection got disconnected and stopped the scp command from completing the process of copying files from remote server to local server.  If you don’t have a lot of files to copy, then you should be able to use the same scp command to copy the same files again until everything got copied from remote server to local server.  What if you got huge amount of files (i.e., in tens of Gigabytes) to copy down from remote server to local server?  The disconnection of the Internet during scp process is a devastation in this situation, because the scp command would restart the copying of existing files that already downloaded to your local server.  This would be a waste of time.

No sweat.  I got a solution for you.  Try to use rsync command to sync remote files to local files instead.  This would mean existing files will be skipped, and rsync would only download new files from remote server to local server.  Of course you can reverse the direction of file copying too such as from local server to remote server using rsync.  Nonetheless, the command right after this paragraph shows you how to stop wasting time and continuing the copying of files from remote server to local server in the case scp got interrupted.

[rsync -avzhe ssh* ~/Download/backup/]

This rsync command I’d mentioned above uses the e parameter to append the ssh command so rsync can be done through SSH for secure file copying.  Basically, the e parameter specifies a remote shell to be used.  By the way, the other parameters are -a (equivalent to -rlptgoD – meaning preserving more files attributes than just using -r), -v (verbose printout), -z (compress files during transfer for faster file transfer), and -h (output numbers in human readable format).  By using rsync this way, you can now continuing the process of copying files from remote server to local server when scp failed to complete the job the first time around.


Installing Latest Version Of Rsync On Mac OS X Snow Leopard By Using MacPort

Did you notice that Rsync on Mac OS X Snow Leopard is version 2.6.9, but the latest version of Rsync is 3.0.8?  Since some people might want to use Rsync between Mac OS X Snow Leopard and Linux, but their Rsync program on Mac OS X Snow Leopard might stutter (i.e., hanging and stop working in the middle of transferring files).  This is why you need to upgrade/update your Rsync to the latest version, but Rsync program on Mac OS X Snow Leopard is a modified version, and so you won’t be able to upgrade/update it unless you use Macport.

So, what is MacPort?  I think Macport is a package manager outside of Apple’s realm, and MacPort tends to repackage Linux/Unix software and make these software available for Mac users.  Rsync is obviously tied to Linux/Unix.  Anyhow, installing MacPort can be really easy, because you can download the dmg file of MacPort and double click dmg file to install MacPort.  The hard part is that MacPort requires you to have XCode, but XCode 3 is available for free on Apple’s website.  XCode 3 is huge, and so you should expect your download will be around 4 GB or so.

I don’t know what version of Rsync that Mac OS X Lion is using, and so this blog post is specifically address Snow Leopard only!  In case Mac OS X Lion isn’t using the latest version of Rsync too, then you can follow this post and try install MacPort.  For your information though, you can use XCode 4 for Mac OS X Lion, because XCode 4 is a newer version of XCode and it’s freely available to download for Mac OS X Lion.  I wish XCode 4 would work with Snow Leopard, but that isn’t the case.  Anyhow, my apology for being digressed.

After you have MacPort installed, you need to do an update of your MacPort by trying this command inside terminal [port -v selfupdate].  Now, you should be able to install latest version of Rsync by trying this command in terminal [port install rsync].  Notice, do not type the square brackets inside the terminal when you execute the commands above.  I place the brackets to discern the commands from regular texts in this blog post.

You should restart your Mac OS X Snow Leopard after the installation of Rsync from MacPort has finished.  After re-logging into your Mac OS X Snow Leopard, you can use the latest Rsync by trying this command [/opt/local/bin/rsync –version].  Notice, that command will show you what version your Rsync is and also the command itself reveals where your latest Rsync is found.  Mac OS X Snow Leopard’s Rsync is found in /usr/bin/rsync, and whenever you try to type [rsync] by itself inside your terminal — Mac OS X Snow Leopard will use the older version of Rsync which is located at /usr/bin/rsync.  Of course, that is not what you want, right?  So you should try to use Rsync from /opt/local/bin/rsync, OK?

Hope this blog post help some people who struggle with Rsync on Mac OS X Snow Leopard!  Enjoy!

Backing Up Mac Using Rsync, Sidestepping Time Machine

Dear Mac users, Apple’s Time Machine is fine and dandy, but some people may not feel so happy about it!  If you’re one of them, I’m happy to tell you that you can use rsync to backup your data.  As on Linux, Mac does utilize rsync and it works the same way as on Linux.  After you rsync whatever data or everything on Mac to an external storage location such as another computer, whenever your Mac has a hard drive failure, you can rest assure all your data can be restored.

I think restoring Mac using rsync is as simple as firing up rsync command again.  You just have to make sure Mac’s OS X is reinstalled onto the replacing hard drive, and then rsync it with your backup.  Although the method is a little crude just to get rsync in a working condition, but as long you have rsync working you can be rest assure that all your backup data will be restored in pristine condition.  For your information, rsync comes by default inside all Mac OS X and Linux, but if I’m wrong you can always download it at a reputable source that you may find through the use of Google.

Using rsync is simple as telling the source to sync with the receiver.  Here is one example:  Let say your Mac has a hard drive failure, you replace the hard drive, reinstalling Mac OS X, firing up a terminal, and then you can begin to type in the rsync command such as [rsync -avz -e ‘ssh -p 1212’ –stats –progress username@remotehost.tld:/home/user/backups/ /].  You can safely ignore the square brackets when entering the rsync command since I put them there for visibility.  Make sure you replace -p 1212 with a correct port number that your ssh server is accepting.  Also, you need to replace the username@remotehost.tld with a working username and IP address or domain name of a remote storage/host.  If my memory serves me well, rsync will not work when you rsync to an unavailable directory (i.e., /home/user/backups/) — it’s imperative for you to create a backups directory at receiver first before firing away the rsync command.

The command I’d mentioned above, it’s for you to restore your data from a remote backup storage.  It’s obvious that you have to reverse the roles of the source and the receiver if you’re going to backup your data to a remote storage.  Here is one way of how to reverse the command above, [rsync -avz -e ‘ssh -p 1212’ –stats –progress / username@remotehost.tld:/home/user/backups/].  Once again, just ignore the square brackets.

There you have it, not only rsync is available on Linux but it’s also available on Mac so you can use it to backup your Mac’s data.  Rsync requires very little resource and so it will be light on your system.  The best part is that you can rsync your Mac with a Linux PC!  The idea is great since not everyone has an extra Mac around, but many more people may have an extra PC/laptop around which utilizes Linux; if Linux isn’t available on such a system, it’s only a download away since Linux is free!

I love it when my Mac is miles away from my Linux, and yet I’m able to rsync at zero cost.  I don’t need to buy an extra external storage or pay for a third party service to backup my data — I could be wrong but Time Machine requires you to either have another Mac or an external storage to be hooked up to your Mac the whole time for it to work.  Last thing on my mind in relating to this very topic, even better, you can virtualize your Linux with VMWare or VirtualBox so you don’t have to sacrifice your real operating system such as Windows 7.