Recently in Backups and storage Category

I've enjoyed Zack Whittaker's iGeneration blog on ZDNet for a while now, but his most recent entry really hits you where you live — if you have a lot of stuff on a computer hard drive that you'd really not like to lose in an instant.

Zack accidentally kicked out the power plug on his computer, scrambled his hard drive ... and faces a huge, huge bill if he wants to recover years' worth of data:

Through no will of trying, I have now come to a sound, unfortunate conclusion, that my hard drive is well and truly screwed. My data is fully intact but I have absolutely no way of accessing it. And did I back up my machine? Of course I did – but on a separate partition, and on that hard drive.

A backup on the same drive? Bad idea.

That's why we all should ignore the garment-rending in the geek arena over the privacy implications of storing data in the cloud with a service such as Amazon S3 (via something like JungleDisk) or with DropBox, Mozy or any number of competing services.

Yes, there are problems in the cloud and with off-site, networked backups. But the benefits far, far outweigh the risks of not doing multiple backups in multiple ways and at multiple locations.

By all means, follow Zach's advice and back up to a detachable, external hard drive (preferably more than one). But don't think that's enough.

You don't just need a backup. You need a backup plan — one with the kind of redundancy that a cloud backup can add to one that also includes multiple hard drives stored in different places.

I'm getting increasingly comfortable keeping and viewing all of my photos online. Printing them? Yeah, every once in a while, but not every one, multiple times, like we used to.

I've got thousands of e-mails, most of which I probably could lose without tears but which serve as a huge database of information that I'd rather preserve than do without.

I probably should be making printouts of my "important" writing, should I do anything like that at some point in the future. Yep, paper — though bulky — can be the ultimate backup. Clay tablets are good, too.

But the reality is that my aversion to clutter (yes, despite my immersion in it) means keeping more and more things on bigger and bigger hard drives (I just saw a 2TB model from Seagate ... 1.5 TB is old hat, I guess) and making sure, through multiple backups (and backups in the cloud that are worth paying for), that I don't lose all of this data due to my own propensity for kicking out power plugs, or any number of natural and man-made disasters.

As an experiment, I decided to bring my Evolutionary Computing presentation on making the journey into free, open-source software — a slide show originally created in OpenOffice Impress 2.4 — into Google Docs, which happens to have a presentation app in addition to the better-known Docs and Spreadsheets components.

I revised the presentation — taking some things out, adding others and providing some updates on what I'm doing — and output it as a PDF.

Download that PDF for your reading pleasure by clicking on the image above or the link below:

Evolutionary Computing (revised July 2009)

Interesting note: I believe that no previous entry on this blog has been filed under so many categories. (And I've been considering dumping Categories entirely and just using tags ...)

I'm no coding guru. And I feel like having to write my own scripts to get stuff done in Unix/Linux is all too much like reinventing the wheel.

Be that as it may, I hacked together these two short scripts to back up my /home files in Ubuntu 8.04 to an external USB drive. I put the scripts in /usr/local/bin and made them executable. I'm lazy enough that I used the Nautilus file manager to do this.

I run the scripts with sudo, meaning in my user account, I open a terminal and do this, entering my password when prompted:

$ sudo usb-backup

$ sudo usb-backup-exclude

For the second script, I created an "exclude file," which the script uses to exclude whichever directories or files I wish. In this case I use it to exclude the .gvfs directory, which breaks the script (and doesn't need to be copied anyway) and in this case to exclude my Thunderbird mail files, since they take so damn long to back up that doing it every day is something I'm not fond of. The beauty of the exclude file is that I can modify it while keeping the script the same.

I'm sure there are many of you who can do and have done a better job than this, but these two scripts appear to work, and that's what counts for me anyway.

There are some pounded-out notes for the scripts; feel free to remove them. They won't affect how the scripts work.

Here are the scripts:

usb-backup:

#! /bin/bash
# Use rsync to back up the /home folder to a 4 GB USB flash drive
# --delete allows for deletion of files on the backup that have been previously deleted on the source drive
# using --exclude to keep rsync from trying to back up ~/.gvfs
# Finally able to remove --ignore-errors now that .gvfs is excluded
rsync -av --delete --exclude 'home/steven/.gvfs' /home /media/disk/ubuntu
exit 0

usb-backup exclude:

#! /bin/bash
# Use rsync to back up the /home folder to a 4 GB USB flash drive
# --delete allows for deletion of files on the backup that have been previously deleted on the source drive
# setting up an exclude file to back up some directories and not others.
# Finally able to remove --ignore-errors now that .gvfs is excluded
rsync -av --delete --ignore-errors --exclude-from '/home/steven/Documents/shell_scripts/exclude' /home /media/disk/ubuntu
exit 0

And here is my "exclude" file which, as you can see from the script above, lives at /home/steven/Documents/shell_scripts/exclude:

/home/steven/.gvfs
/home/steven/.mozilla-thunderbird

.gvfs note: I've done similar scripts before in OpenBSD and Debian, and I don't believe either used the GNOME Virtual Filesystem, so there was no need to exclude ~/.gvfs when using rsync.

I've been bringing more data into my main Ubuntu 8.04 LTS installation on one of my two Toshiba Satellite 1100-S101 laptops, and I continue to be satisfied with the performance of what by most accounts is the world's most popular desktop Linux distribution.

No, its GNOME desktop isn't as fast as Debian's. But even though I do have Xfce (and not the full Xubuntu) installed on this Ubuntu laptop, I'm still using the brownish-themed GNOME that ships with the distro.

I'm getting used to all the GNOME-ish touches in the Nautilus file manager and in Ubuntu/GNOME in general that makes a full-fledged desktop environment such a nice place to work.

After planning for weeks to take my main production laptop from OpenBSD 4.4 to 4.5, I sweated through the upgrade only to lose what was perfect X compatibility and pull the "kill switch," which in this case was transferring everything in my freshly rsync'd backup to my identical Toshiba Satellite 1100-S101 laptop running Ubuntu 8.04 LTS, a system I've been running for quite awhile on this and another laptop — and which has thus far proven itself to be stable enough for the pounding I give these machines in my daily work.

OpenBSD 4.4 basically "saved" me and one of these marginal Toshiba laptops (both were destined for the garbage) last November when I could barely get an install CD of any type to boot. The install floppy in OpenBSD enabled me to quickly set up a system that worked quite well and did almost everything I needed it to do. And stability was almost a given. I rarely had a problem that wasn't inherent to OpenBSD itself (such as the difficulty of installing Java, nothing past Flash Player 7, the extra steps required to properly configure things such as CUPS).

Since the system ran so well — just like Ubuntu 8.04, video on this Intel-based system ran perfectly with no xorg.conf — I kept it going for the entire six months of the OpenBSD 4.4 release's life.

As those who use OpenBSD know, upgrading the operating system is not as easy as it is in your average Linux distribution. It pretty much comes with the territory that a -release upgrade requires preparation, following instructions, and a bit of manual command-line work. Many times I've heard — both in OpenBSD and in Linux for that matter — that it's easier and cleaner to do a full reinstall rather than an in-place upgrade.

I will still try a full reinstall of OpenBSD 4.5. And I'd like to try running -current — the OpenBSD development branch that can be regularly updated and which is famously stable despite the "development" tag.

But right here, right now, I can't spend weeks diagnosing my X issues (briefly, there's some funky junk hanging from the cursor, and "artifacts" linger on the screen, which isn't redrawn fast enough/often enough to make X usable). The same thing turned me away from Debian Lenny on this and my Gateway Solo 1450 laptop in the months before the then-Testing distro went Stable. Because of my affection for Debian (still one of my very favorite operating systems), I spent weeks trying to diagnose the problem before realizing that dozens of other distros relieved me of the need to obsess (unsuccessfully) over it.

Right now the Gateway, used by our 5-year-old dual-boots Ubuntu 8.04 for her and CentOS 5.3 just because it runs so extremely well on that particular laptop.

And for months now I've had this other Toshiba laptop running Ubuntu 8.04 as a backup. I have Java installed, which I do need. Flash, too. The Opera Web browser.

Today I added Inkscape, Thunderbird, gFTP and Gparted.

On the OpenBSD laptop, I had about 1 GB of e-mail in Thunderbird. It makes rsyncing the box such hell that I'm thinking of writing a script that EXCLUDES the Thunderbird files just so the rest of the backup doesn't take so damn long ... but I digress.

I figured out how to bring my Thunderbird settings and mail over to the Ubuntu machine. I did the same with my Firefox bookmarks.

-- Begin tutorial:

Moving bookmarks from one Firefox 3 installation to another:

• Since Firefox now uses the SQlite database to store/organize its bookmarks, simply moving the bookmarks.html file from one Firefox 3 installation to another will DO ABSOLUTELY NOTHING. You need to do it another way, which I describe right here. First, grab the bookmarks.html file from your old FF installation and put it somewhere in your /home directory where you can easily find it.

• In the Firefox 3.0 installation where you want to IMPORT the bookmarks, go to the Bookmarks tab and click on/choose Organize Bookmarks.

• Click on the Import and Backup drop-down menu and click Import HTML.

• Then navigate to the bookmarks.html file from your old FF 3 installation (you have moved it over already, haven't you?) and click it to bring it into your new installation.

• Note: In Ubuntu at least, this process WON'T allow you to see hidden files or directories, so before you begin, copy your old bookmarks.html file to a place in your home directory where you don't need to go into your old installation's .mozilla directory, for instance.

• FYI: In both of my Firefox 3 installations, the bookmarks.html file is located here:

  /home/username/.mozilla/firefox/xxxxxxxx.default/bookmarks.html

  In the above example, "username" is your actual username, and the eight x's are the unique alphanumeric prefix that Firefox gives to your "default" directory under /.mozilla/firefox/
  

-- End tutorial.

-- Resume rant.

OK, so I'm fully operational in Ubuntu at this point. My respect and admiration for the developers and users of OpenBSD remains, and I hope to get the other Toshiba fully operational under OpenBSD 4.5 as soon as possible.

But I'd be lying if I didn't say I was relieved to have, in Ubuntu, a machine and system that easily updates all of its software with a few clicks and provides me with what — at this point — is a trouble-free working environment.

Of course that could all change. I'll see over the next week how well Ubuntu 8.04 LTS performs on this hardware, with my chosen applications and for the tasks I have.

I could start the distro-hopping merry-go-round and go back to Debian, try out Slackware, ZenWalk, etc., but right now if Linux in this form does what I need it to do (not crash, run acceptably fast, wash, rinse, repeat), I'll be sticking with Ubuntu as long as it fills the bill.

This week's Tech Talk column covers the creation of what I call The Self-Reliant Thin Client, which is basically a very-bare-bones PC that is booting and running off of a Compact Flash module instead of a traditional spinning hard drive.

Here is the photo gallery, which will get full captions when I get the time to write them.

I have been wanting to test solid-state storage technology for some time now, and with the solid-state drive option for Mac laptops costing $600 (over and above the MacBook's $1,600 price), the drives themselves as laptop replacements in 64 GB sizes going for $170, I decided to use the slower but way cheaper Compact Flash technology, which is very common in high-end digital cameras.

I finally got an 8 GB Compact Flash chip from newegg.com for about $20, and I'm backing up my user files on a USB flash drive plugged into the back of the box.

The box — which started out as a Maxspeed Maxterm thin client — is running Debian Etch.

Since the Power Macintosh G4/466 has enough space for four hard drives, I decided to put a second one in the box.

I originally planned to dual-boot Debian Etch and OpenBSD, but I still can't get the system to even boot OpenBSD after an install, so I abandoned that plan. After two successful OpenBSD installs on i386, I figured I could handle a MacPPC install, but it was not to be.

And since Debian Etch installs — and runs — so well, I'm keeping it.

Before the last unsuccessful OpenBSD install, I set the second drive's jumpers to make it a "slave." Then I used an extra IDE cable as an "extension cable" so I could plug the second IDE input from the too-short motherboard cable into the second drive.

I'm pretty accustomed to chaining IDE cables together for longer runs — I did it for over a year with my VIA thin client.

Anyway, I booted into Debian Etch, ran the GNOME Partition Editor to create an ext3 filesystem on the backup drive, then mounted it and did a few tests with rsync.

Yep, I'm using rsync to do the backups. I first learned about it in Carla Schroder's great "Linux Cookbook" (which all of you should own ... and which O'Reilly should beg Carla and pay her well to revise immediately).

Rsync is in the default Ubuntu install, but in Debian Etch, you have to add it:

# aptitude install rsync

In order to use the backup drive, I created a directory called hdb1 in /media and used it to mount the drive:

# mkdir /media/hdb1

Instead of mounting it from the command line, I put a line in /etc/fstab.

Here's what /etc/fstab looks like on this machine:

# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/hda3 / ext3 defaults,errors=remount-ro 0 1
/dev/hda5 /home ext3 defaults 0 2
/dev/hda4 none swap sw 0 0
/dev/hdb1 /media/hdb1 ext3 defaults 1 2
/dev/hdc /media/cdrom0 udf,iso9660 user,noauto 0 0

I made a directory called homebackup on /media/hdb1, which is where my backups of /home will go:

# mkdir /media/hdb1/homebackup

To run the backup from the command line, here's what I did (at a root prompt):

# rsync -av --delete /home /media/hdb1/homebackup

(A note on rsync: The FIRST time I ran rsync with the line above, a home directory was created in /media/hdb1/homebackup. Subsequent uses of the same rsync command just update the directory; they don't create it again.)

The beauty of rsync: I really didn't explain rsync all that well. Basically, it's a great backup utility that, when used multiple times, only backs up the changes between the source and the destination directories.

What the switches mean in rsync -av --delete:

-a stands for archive mode. It is a catch-all switch that turns on a variety of things in rsync, most of which I don't understand. I use all of these switches because Carla Schroder told me to. For more (or less) clarity, check out the man page for rsync.

-v stands for verbose so rsync outputs to the console exactly what it's doing.

--delete tells rsync to delete files in the destination directory that have been previously deleted from the source directory. Without this switch, the backup directory would just get bigger and bigger. You might want that; without --delete, everything you delete would still be in the backup directory. But I don't, so I'm using this switch.

in order not to have to remember to type my entire rsync command with the switches and the directory information, I put it in a very simple shell script in /usr/local/bin.

I called the shell script backup — can't forget that.

Here's what is in the /usr/local/bin/backup file:

#!/bin/bash

# This file backs up the /home partition to the secondary drive with rsync
rsync -av --delete /home /media/hdb1/homebackup

To make the file executable, I ran:

# chmod a+x /usr/local/bin/backup

I'm not a big fan of using su to root, so I added my user account to the sudoers file (after I su'd to root):

# visudo

When you invoke visudo from a root prompt, the system opens the sudoers file with the default editor. Usually that editor is vi, but in Debian, the default editor, as configured, is nano. I've been using vi a lot lately in OpenBSD (with the i386 systems on which I can actually successfully install it). But for me it's easier to use nano, with it's F3 to save, CTRL-x to exit, and just moving the cursor around with arrow keys and not having to worry about going from command mode to insert mode as in vi.

Now that my little vi rant is over, here's what I did in the sudoers file after visudo brought it up in nano:

I gave my user account the same privileges as root. I wouldn't do this for other users; I'd figure out how to more finely grain the permissions, which is what sudo is really good at. But I'm not running a huge multiuser box, so giving my user account those privileges is good enough for now.

Here's what the relevant section of the sudoers file looks like:

# User privilege specification
root ALL=(ALL) ALL
steven ALL=(ALL) ALL

Back to the backups: This script will run without sudo or su privileges, but the --delete switch will not work because the non-root user doesn't have privileges in the lost+found directory in /home. So if you want to back up the entire /home directory, including all the files for all users, you should run it this way:

$ sudo backup

or after su to root:

# backup

Of course, if you want to set this up for your individual users, you can rewrite the rsync line to only back up their own /home files:

rsync -av --delete /home/steven /media/hdb1/homebackup

Then they could run it from a regular prompt, without root privileges, and it will work.

Since I'm the only user of this box, I made the rsync command cover the entire /home folder, which means, for me, using sudo to make it work.

What about a cron job? Sure, you could do this as a cron job, but I'm not in the habit of leaving the box on all the time, so cron (or even anacron) doesn't fit in with my computing habits. I guess anacron, which runs jobs after a certain interval of time has passed (and not only at certain, specified times), would work, but I'd rather just run the shell script periodically and see the jobs scroll down my terminal window.

But like anything in Unix, there's more than one way to skin any given cat.

Ubuntu note: In Debian, the "main" user does not automatically have sudo privileges. But in Ubuntu, that user does have sudo ability. You don't have to use visudo at all for that user; just use sudo from the get-go. I'm not sure about subesequent users in Ubuntu, but I sure hope they don't have sudo privileges by default; I think they don't but I don't have a Ubuntu box handy to check.

The trade-off for Ubuntu users is that they can't su to root. Well, they can ... Ubuntu encourages you to use sudo, but you can get a root shell this way:

$ sudo su

After you type in your password, you will have a root prompt:

#

As I said, I'm not a fan of using su to root, and I'd rather use sudo, but there are some things that sudo can't do, and that's when the Ubuntu su trick is necessary. I'd like to thank whoever it is who passed that one along to me.

Conclusion: If you hae a place to back up your files, using rsync is a great way to make those backups. Rsync also excels at backups over SSH, which Carla Schroder goes into in great detail in her book. She also shows you how to set up an rsync server.

Just buy the book already, will ya?

Robin Harris of ZDNet's Storage Bits blog always has excellent advice on backups and the best ways to do them. Here he is with How to archive on disk drives, which explains why putting something on a hard drive and then powering it off for extended periods of time isn't such a great idea:

The reason: the bits on the hard drive will gradually lose their magnetism, sometimes in as little as 12 months. Disks automatically rewrite marginal data blocks on the fly.

That doesn't happen when the disk is powered off.