Recently in rsync Category

I'm a stickler for backups. I do them all the time. I make more than one copy.

The worst thing about my backups is Thunderbird mail. I POP the mail down to my local computer for various reasons, and if you use Thunderbird this way, you know that it uses the mbox format, which leads to some very large files.

And every time you get an e-mail, those large files change ever so slightly.

I'm using rsync as my backup utility, and backing up the Thunderbird mail always takes a very, very, very long time.

I don't know for sure if I'm invoking the "copy only parts of the file that changed" switch in rsync (I use rsync -av --delete), but whether I am or not, it takes an awfully long time to complete the backup.

It takes so long that I have one script to back up all my user files except Thunderbird mail and another script that does Thunderbird mail only.

All in all, I'd rather have my POP mail saved in a format that creates thousands of small files, most of which never change, rather than a few dozen huge ones that are always changing.

As an experiment, I decided to bring my Evolutionary Computing presentation on making the journey into free, open-source software — a slide show originally created in OpenOffice Impress 2.4 — into Google Docs, which happens to have a presentation app in addition to the better-known Docs and Spreadsheets components.

I revised the presentation — taking some things out, adding others and providing some updates on what I'm doing — and output it as a PDF.

Download that PDF for your reading pleasure by clicking on the image above or the link below:

Evolutionary Computing (revised July 2009)

Interesting note: I believe that no previous entry on this blog has been filed under so many categories. (And I've been considering dumping Categories entirely and just using tags ...)

I'm no coding guru. And I feel like having to write my own scripts to get stuff done in Unix/Linux is all too much like reinventing the wheel.

Be that as it may, I hacked together these two short scripts to back up my /home files in Ubuntu 8.04 to an external USB drive. I put the scripts in /usr/local/bin and made them executable. I'm lazy enough that I used the Nautilus file manager to do this.

I run the scripts with sudo, meaning in my user account, I open a terminal and do this, entering my password when prompted:

$ sudo usb-backup

$ sudo usb-backup-exclude

For the second script, I created an "exclude file," which the script uses to exclude whichever directories or files I wish. In this case I use it to exclude the .gvfs directory, which breaks the script (and doesn't need to be copied anyway) and in this case to exclude my Thunderbird mail files, since they take so damn long to back up that doing it every day is something I'm not fond of. The beauty of the exclude file is that I can modify it while keeping the script the same.

I'm sure there are many of you who can do and have done a better job than this, but these two scripts appear to work, and that's what counts for me anyway.

There are some pounded-out notes for the scripts; feel free to remove them. They won't affect how the scripts work.

Here are the scripts:

usb-backup:

#! /bin/bash
# Use rsync to back up the /home folder to a 4 GB USB flash drive
# --delete allows for deletion of files on the backup that have been previously deleted on the source drive
# using --exclude to keep rsync from trying to back up ~/.gvfs
# Finally able to remove --ignore-errors now that .gvfs is excluded
rsync -av --delete --exclude 'home/steven/.gvfs' /home /media/disk/ubuntu
exit 0

usb-backup exclude:

#! /bin/bash
# Use rsync to back up the /home folder to a 4 GB USB flash drive
# --delete allows for deletion of files on the backup that have been previously deleted on the source drive
# setting up an exclude file to back up some directories and not others.
# Finally able to remove --ignore-errors now that .gvfs is excluded
rsync -av --delete --ignore-errors --exclude-from '/home/steven/Documents/shell_scripts/exclude' /home /media/disk/ubuntu
exit 0

And here is my "exclude" file which, as you can see from the script above, lives at /home/steven/Documents/shell_scripts/exclude:

/home/steven/.gvfs
/home/steven/.mozilla-thunderbird

.gvfs note: I've done similar scripts before in OpenBSD and Debian, and I don't believe either used the GNOME Virtual Filesystem, so there was no need to exclude ~/.gvfs when using rsync.

(Yes, I do have the OpenBSD T-shirt with this design. It doesn't get more geeky.)

I'm getting ready to give the $0 Laptop (Gateway Solo 1450) to our daughter to run her educational games (Childsplay, Gcompris, TuxPaint) on Ubuntu Hardy with the non-crashing Xfce window manager instead of the crashy version of GNOME in this Ubuntu build.

To replace that machine for me, I pulled a Toshiba Satellite 1101-S101 laptop from the boneyard.

With a 1.3 GHz Celeron processor, 248 MB RAM (how it has this amount, I don't know) and a 20 GB hard drive, the specs are pretty similar to the Gateway, except for the Gateway's 1 GB of memory, which I'll probably split between the two machines.

The Toshiba came to me with Windows XP, and this time I wanted to preserve Windows and dual-boot it with a FOSS OS. The CD/DVD drive is extremely flaky. I think it's dying. It does better with "commercial" CDs, and I did get it to boot Partition Magic so I could shrink the NTFS Windows partition and set it up for Linux.

The only Linux CD I could boot was Debian's Etch and a Half. Something was squirrely on our network, and I couldn't get DNS working in the installer. I could've done a minimal install, fixed /etc/resolv.conf and then brought the rest of Debian into the box, but I took this opportunity to go in a different direction.

With all the CPU fan issue on the Gateway, I could never run OpenBSD (or NetBSD or even FreeBSD after the first boot) because I couldn't get the noisy CPU fan under control.

I powered up the Toshiba, which couldn't get networking in Windows either. Since I don't yet have the administrator password, I couldn't update the DNS settings.

I went to an OpenBSD mirror and downloaded a floppy image plus a DOS/Windows utility that helped me create a bootable OpenBSD install floppy. (Before anybody mentions this, I know I could've just as easily created a Debian boot floppy.)

The Toshiba successfully booted off the OpenBSD floppy, and I was able to plug in a mirror and do a full install over the network.

This was my first dual-boot install of OpenBSD, and after the install was done, the machine wouldn't boot at all. I hadn't installed a bootloader and thought the box would boot into Windows, where I planned to modify that bootloader to choose between Windows XP and OpenBSD. Instead I got a "no operating system" message.

And I don't have a Windows XP disc from which to "repair" the master boot record.

So I rebooted with the OpenBSD floppy, dropped down to a shell and added the OpenBSD bootloader at the prompt:

# fdisk -u wd0

Then I rebooted and was in OpenBSD. There is a GRUB package for OpenBSD, and I'll probably install that so I can easily dual-boot either Windows and OpenBSD or eventually Linux and OpenBSD. There are other alternatives as far as bootloaders go, but my familiarity with GRUB is what is governing my decision in this case.

I'm also going to add rsync as well. I have no skills when it comes to OpenBSD's dump and restore utilities, so having rsync is another plateful of Linux-like comfort food that will help me get along in OpenBSD.

Other packages I've installed thus far: nano, mc (the Midnight Commander file manager), Rox-filer (my favorite X file manager), Geany (X text editor) and the Firefox (I probably should've gotten the version with Java, but I'm going to try to add the Java developer's kit and get the Java runtime that way) and Opera Web browsers.

Opera came via a port and not a precompiled package, and it took a lot longer to install this time than the last time I installed it in OpenBSD (on the Compaq Armada 7770dmt), if I recall correctly.

When you download the ports tree and install from there, everything is fetched for you and compiled when needed. Looking at all the output in the terminal, it looks like these ports could never work, but in my experience with OpenBSD they always do. This time was no different. It took maybe 45 minutes to get all the dependencies plus Opera, but after that it worked immediately.

I've grown accustomed to OpenBSD's default window manager, Fvwm2, and I'll probably stick with it for at least awhile before adding any others. Unlike Debian, Ubuntu, Slackware, etc., installing an app in OpenBSD doesn't automatically update the menus, so you have to manage this yourself. Getting into the guts of the .fvwmrc file is more instructive than not, and once I figured out how to do it, it got less arduous.

I still don't like waiting for ports to download, compile and install, so having 4000+ precompiled packages for i386 is a very good thing.

After a year of strugging with and complaining about the Gateway fan blasting away under OpenBSD, I couldn't believe that I was running OpenBSD 4.4 on the Toshiba with no CPU fan problem whatsoever. Everything from autoconfiguration of my two network interfaces (one Realtek 8189 wired Ethernet, the other an Orinoco WaveLAN PCMCIA wireless) to a perfect xorg.conf made this OpenBSD install go .

I haven't checked audio yet, but I've never had OpenBSD fail to configure the sound card.

I've always read that most OpenBSD developers use laptops to code in the OS, and now that I have this Toshiba running OpenBSD better than anything I've tried before, I'm amazed at how well it installs and runs on this specific platform.

I've probably written a half-dozen posts about exactly why I'm running OpenBSD, and I'll probably write another one as time allows in the week ahead.

And I'll be either ordering a CD set or contributing directly to the OpenBSD project in the days ahead.

Since the Power Macintosh G4/466 has enough space for four hard drives, I decided to put a second one in the box.

I originally planned to dual-boot Debian Etch and OpenBSD, but I still can't get the system to even boot OpenBSD after an install, so I abandoned that plan. After two successful OpenBSD installs on i386, I figured I could handle a MacPPC install, but it was not to be.

And since Debian Etch installs — and runs — so well, I'm keeping it.

Before the last unsuccessful OpenBSD install, I set the second drive's jumpers to make it a "slave." Then I used an extra IDE cable as an "extension cable" so I could plug the second IDE input from the too-short motherboard cable into the second drive.

I'm pretty accustomed to chaining IDE cables together for longer runs — I did it for over a year with my VIA thin client.

Anyway, I booted into Debian Etch, ran the GNOME Partition Editor to create an ext3 filesystem on the backup drive, then mounted it and did a few tests with rsync.

Yep, I'm using rsync to do the backups. I first learned about it in Carla Schroder's great "Linux Cookbook" (which all of you should own ... and which O'Reilly should beg Carla and pay her well to revise immediately).

Rsync is in the default Ubuntu install, but in Debian Etch, you have to add it:

# aptitude install rsync

In order to use the backup drive, I created a directory called hdb1 in /media and used it to mount the drive:

# mkdir /media/hdb1

Instead of mounting it from the command line, I put a line in /etc/fstab.

Here's what /etc/fstab looks like on this machine:

# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/hda3 / ext3 defaults,errors=remount-ro 0 1
/dev/hda5 /home ext3 defaults 0 2
/dev/hda4 none swap sw 0 0
/dev/hdb1 /media/hdb1 ext3 defaults 1 2
/dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0

I made a directory called homebackup on /media/hdb1, which is where my backups of /home will go:

# mkdir /media/hdb1/homebackup

To run the backup from the command line, here's what I did (at a root prompt):

# rsync -av --delete /home /media/hdb1/homebackup

(A note on rsync: The FIRST time I ran rsync with the line above, a home directory was created in /media/hdb1/homebackup. Subsequent uses of the same rsync command just update the directory; they don't create it again.)

The beauty of rsync: I really didn't explain rsync all that well. Basically, it's a great backup utility that, when used multiple times, only backs up the changes between the source and the destination directories.

What the switches mean in rsync -av --delete:

-a stands for archive mode. It is a catch-all switch that turns on a variety of things in rsync, most of which I don't understand. I use all of these switches because Carla Schroder told me to. For more (or less) clarity, check out the man page for rsync.

-v stands for verbose so rsync outputs to the console exactly what it's doing.

--delete tells rsync to delete files in the destination directory that have been previously deleted from the source directory. Without this switch, the backup directory would just get bigger and bigger. You might want that; without --delete, everything you delete would still be in the backup directory. But I don't, so I'm using this switch.

in order not to have to remember to type my entire rsync command with the switches and the directory information, I put it in a very simple shell script in /usr/local/bin.

I called the shell script backup — can't forget that.

Here's what is in the /usr/local/bin/backup file:

#!/bin/bash

# This file backs up the /home partition to the secondary drive with rsync
rsync -av --delete /home /media/hdb1/homebackup

To make the file executable, I ran:

# chmod a+x /usr/local/bin/backup

I'm not a big fan of using su to root, so I added my user account to the sudoers file (after I su'd to root):

# visudo

When you invoke visudo from a root prompt, the system opens the sudoers file with the default editor. Usually that editor is vi, but in Debian, the default editor, as configured, is nano. I've been using vi a lot lately in OpenBSD (with the i386 systems on which I can actually successfully install it). But for me it's easier to use nano, with it's F3 to save, CTRL-x to exit, and just moving the cursor around with arrow keys and not having to worry about going from command mode to insert mode as in vi.

Now that my little vi rant is over, here's what I did in the sudoers file after visudo brought it up in nano:

I gave my user account the same privileges as root. I wouldn't do this for other users; I'd figure out how to more finely grain the permissions, which is what sudo is really good at. But I'm not running a huge multiuser box, so giving my user account those privileges is good enough for now.

Here's what the relevant section of the sudoers file looks like:

# User privilege specification
root ALL=(ALL) ALL
steven ALL=(ALL) ALL

Back to the backups: This script will run without sudo or su privileges, but the --delete switch will not work because the non-root user doesn't have privileges in the lost+found directory in /home. So if you want to back up the entire /home directory, including all the files for all users, you should run it this way:

$ sudo backup

or after su to root:

# backup

Of course, if you want to set this up for your individual users, you can rewrite the rsync line to only back up their own /home files:

rsync -av --delete /home/steven /media/hdb1/homebackup

Then they could run it from a regular prompt, without root privileges, and it will work.

Since I'm the only user of this box, I made the rsync command cover the entire /home folder, which means, for me, using sudo to make it work.

What about a cron job? Sure, you could do this as a cron job, but I'm not in the habit of leaving the box on all the time, so cron (or even anacron) doesn't fit in with my computing habits. I guess anacron, which runs jobs after a certain interval of time has passed (and not only at certain, specified times), would work, but I'd rather just run the shell script periodically and see the jobs scroll down my terminal window.

But like anything in Unix, there's more than one way to skin any given cat.

Ubuntu note: In Debian, the "main" user does not automatically have sudo privileges. But in Ubuntu, that user does have sudo ability. You don't have to use visudo at all for that user; just use sudo from the get-go. I'm not sure about subesequent users in Ubuntu, but I sure hope they don't have sudo privileges by default; I think they don't but I don't have a Ubuntu box handy to check.

The trade-off for Ubuntu users is that they can't su to root. Well, they can ... Ubuntu encourages you to use sudo, but you can get a root shell this way:

$ sudo su

After you type in your password, you will have a root prompt:

#

As I said, I'm not a fan of using su to root, and I'd rather use sudo, but there are some things that sudo can't do, and that's when the Ubuntu su trick is necessary. I'd like to thank whoever it is who passed that one along to me.

Conclusion: If you hae a place to back up your files, using rsync is a great way to make those backups. Rsync also excels at backups over SSH, which Carla Schroder goes into in great detail in her book. She also shows you how to set up an rsync server.

Just buy the book already, will ya?