Backups with Restic, A Raspberry Pi, and a 4TB HDD

Hama Rikyu Garden

I had been wanting to try Restic since hearing Alex Neumann talk about it on the Go Time podcast Episode 48. TLDR, it’s just as awesome as I had imagined.

What is it?

Restic1 is an modern command line backup utility written Go. You point it at the data you want to backup, give it a place to store the backup, and provide a password:

  • The data can be anything from a single file to….well, pretty much as much data as you have on your computer.
  • The place you store the backup can be another directory on your computer, a remote server, or a cloud storage service like Amazon S3 or Backblaze B2.
  • The password is used to securely encrypt your data, which is table stakes for any remote backup.

What makes Restic special is its speed and simplicity. It cleanly abstracts the storage backend; it handles indexing, data de-duplication, snapshots, encryption, and backup names automatically; and it implements everything without any hassles or headaches. It just works.

What’s your use case

In my case, I wanted to create a backup of a large external hard drive that holds things like my iTunes Library, my Photos library, and iOS device backups. One way to do this would be to use rsync. Another way on MacOS would be to use Time Machine.2 Restic offers a third option, with the added benefit that the backup itself doesn’t even have to be local. In my case, I decided to backup to a 4TB external hard drive attached to a Raspberry Pi 2.

But wait–isn’t this incredibly slow? Even the cheap 5400 RPM spinning disks in modern external hard drives can read and write data at more than 100MB/s. That’s almost a gigabit of data a second when all the peripherals on the Pi share a single 480Mbps internal USB 2.0 bus.

In fact, it is–the backup progresses at about 9MB/s, which might sound pretty good until you consider that it would take around 5 days to backup an entire 4TB of data. But that’s OK. I don’t actually have 4TB of data to backup, and it would take more than a couple of days for a more powerful single board computer like the ODROID-XU4 or the Asus Tinker Board to arrive. Not to mention a bit more money.

Running Restic

This is the last step, but I’m including it here because it’s so simple, and if you don’t feel like setting up a Raspberry Pi you could just point it at something like Backblaze B2.3

Install Restic

Detailed instructions here. Done and done on MacOS:

brew tap restic/restic
brew install restic

Initialize a new repository

restic --repo sftp:[email protected]:/mnt/backup/restic init
  • sftp:[email protected]
    • We’re backing up to the Raspberry Pi via SSH, so we use the sftp prefix.
    • pi is the remote username on the Raspberry Pi. raspberrypi.local is the local network host name for the Raspberry Pi.4
  • /mnt/backup/restic is the mount point on the Raspberry Pi where we want the backup to be saved.
  • init tells Restic that we want to create a new backup repository. It will ask us for a password and do all the configuration we need

Create a backup

Use this command each time you want to backup your system.

restic --repo sftp:[email protected]:/mnt/backup/restic backup ~/
  • backup: Create a new backup…
  • ~/: …of everything in my home directory

Remove all but the most recent snapshot from the repository

By default, Restic creates a new snapshot of your data every time you run it. In my case, I usually only want to keep the latest one. As you might expect, Restic supports many, many different options for specifying the subset of snapshots to keep.

restic --repo sftp:[email protected]:/mnt/backup/restic forget --keep-last 1 --prune

Repository checkups

Regular housekeeping to make sure the repository is in good shape:

restic --repo sftp:[email protected]:/mnt/backup/restic check

In some situations,5 rebuilding the index may also be necessary:

restic --repo sftp:[email protected]:/mnt/backup/restic rebuild-index

Although both of these commands could be run on the Raspberry Pi direct;y, I found that check tends to eat up a lot of RAM, leading to excessive swapping and hangs when run on the Pi itself. Solution? Just run it remotely from the computer you’re backing up:-)

Setting up the Raspberry Pi

Turns out, I already have a guide for this: How To: Prep A Raspberry Pi 2 For Compiling Swift.

In fact, I already had the same Raspberry Pi I had set up last year. sudo apt-get update, sudo apt-get dist-upgrade, sudo apt-get update and everything was ready to go. Even better, despite having been powered down for 9 months the Pi still had the same IPv6 address when I plugged it in. Remote backups? Done and done!

Attaching a 4TB external drive to the Raspberry Pi

The Raspberry Pi 2 has 4 USB ports, all connected to a single 480 Mbps USB 2.0 bus. While they’re not particularly fast, 4 full-sized USB ports feels somehow abundant and luxurious at a time when most people use laptops, and most laptops have 1, 2, or 3.

As expected, the drive showed up as /dev/sda as soon as I plugged it in.

Linux filesystem redux

For as long I can remember Linux filesystems have been in a state of flux. Just as the ‘Year of Linux on the Desktop’ has always been right around the corner, for about two decades it seems like we’ve been a year or two away from the ideal modern filesystem. As is par for the course, there is still no clear winner. The good news, though, is that there are three pretty good options: Btrfs, ext4, and ZFS.

For this project I was looking for two key properties:

  • Encryption: At some point all disks get discarded, and when they do it can be inconvenient or even impossible to wipe all the data on them. Full disk encryption is like shredding important documents before throwing them away–probably unnecessary, but a good simple step to improve your privacy and protect against identity theft.

  • Bit-rot protection: If any of the 32-trillion bits on the drive get flipped I want them to be repaired seamlessly. Ars Technica has an excellent explanation and demonstration.

The first thing I learned is that encryption is actually a separate concern from the filesystem on Linux. Similar to Core Storage on MacOS, you can use LUKS (actually dm-crypt + LUKS) to create an encrypted volume that you can then format with any filesystem you like.

The requirement for bit rot protection took ext4 out of the running, leaving me with a choice between Btrfs and ZFS:

Faced with two not-terribly-appealing options, I chose Btrfs. It has a good open-source license (BSD), it’s in the Linux kernel, and at the end of the day I’m pretty sure my requirements for this drive are low enough that any filesystem would be fine.

Linux filesystem setup

Get a list of block devices

You’ll probably see the internal SD card reader as mmcblk0. In my case the external 4TB drive ended up as sda. This may differ depending on your configuration.

lsblk

Remove existing partions and create a new one with fdisk

Note: This will destroy everything on the drive.

sudo fdisk /dev/sda # Change sda to the device for your drive

fdisk can be a bit intimidating to work with, because you can easily destroy everything on your computer–or at least your boot record. But we’re here to learn, right? So start it up and type m for help:-)

In my case, I used these commands:

p # Show me what partitions I'm working with
g # Remove everything and start with a blank GPT partition table
n # Create a new partition
[enter] # Start at the start of the drive, the default
[enter] # End at the end of the drive, the default
p # Find out what we've created
q # Quit without saving. Or, w to save everything if it looks good

Create an encrypted volume and a filesystem on top of it

Start by installing LUKS tools. Restart or just load the kernel modules you need with modprobe:

sudo apt-get install cryptsetup
sudo modprobe dm-crypt sha256 aes

Create an encrypted volume on sda1 with cryptsetup:

Note: Unfortunately, I cannot say whether the command below will actually create a volume that would be considered ‘securely’ encrypted. While the default cipher (AES) is generally considered to be very secure, poor configuration choices could leave you with a volume that’s no more secure than one that isn’t encrypted at all. The options below are what I used; they may be more or less secure than the defaults. 6 In any case, choose a very long and completely un-memorizable passphrase. Save it in your password manager.

sudo cryptsetup -v --hash sha256 --iter-time 10000 --use-random luksFormat /dev/sda1

See what we’ve created:

sudo cryptsetup luksDump /dev/sda1

Unlock the encrypted volume as backup:

sudo cryptsetup luksOpen /dev/sda1 backup

Create a Btrfs filesystem on the encrypted volume, and mount it at /mnt/backup:

sudo mkfs.btrfs /dev/mapper/backup
mkdir -o /mnt/backup
sudo mount /dev/mapper/backup /mnt/backup/

Annnnd….that’s it! You’re ready to run Restic (see the section above).

In the interest of convenience one may want to configure LUKS with a key so that it can be automatically mounted at boot time. Since the key itself has to be stored somewhere, however, this defeats part of the purpose of using encryption in the first place.

Getting the external hard drive to spin down

After several days, you may notice that your external hard drive doesn’t spin down automatically. Fortunately, there’s an excellent guide to making that happen on Maker-Tutorials. These are the commands I used:

Check the power status of the drive

sudo hdparm -C /dev/sda

Update the hdparm configuration

Start by getting the external drive’s UUID:

blkid

Note: Use the UUID of the encrypted partition on the drive (/dev/sda1 on my system), not the UUID of the unlocked volume.

Then add the following at the end of /etc/hdparm.conf, replacing 01234567-89ab-cdef-0123-456789abcdef with the actual UUID of the drive:

# External HD
# 60 is 5 minutes
/dev/disk/by-uuid/01234567-89ab-cdef-0123-456789abcdef {
	spindown_time = 60
}

And finally, reload the updated configuration file:

sudo systemctl reload hdparm

Monitoring the Raspberry Pi while Restic is running

Dstat is a great tool for monitoring CPU, IO, and network traffic over the course of hours or days. Just set it to log a line every minute and leave it open in a separate SSH session.

sudo apt-get install dstat
dstat 60

dstat running on a Raspberry Pi 2 during a Restic backup from a remote host

I also often keep a window open watching systemd logs:

sudo journalctl -f

As usual, I also recommend htop for watching CPU and memory usage of in realtime. In my case, my Raspberry Pi 2 keeps most of its cores pretty busy with SSH and kernel processes while Resic runs.

Conclusion

By far the easiest part of this process was using Restic itself. And that’s a good thing, because Restic can backup to any number of services–everything from a drive plugged directly into a computer to a commercial cloud service like Backblaze’s B2.

In my case, a home Raspberry Pi server turned out to be the best solution for my needs. In fact, it worked so well that I found myself backing up my MacBook and a friend’s laptop as well.7

Next step? Backup my local Restic backup to Backblaze B2, perhaps with Restic.

Updates

  1. 2017/8/18 Initial post published
  2. 2017/8/24 Added section about getting the external hard drive to spin down

  1. As a proper noun, I capitalize Restic except when it appears in code as the name of the command itself (restic). [return]
  2. While I have done this in the past, I had some issues creating a backup from one external drive to another this time. [return]
  3. Or, well, just sign up for Backblaze’s consumer product and use that to backup your computers. [return]
  4. raspberrypi is the default host name in Raspbian. When you add .local your Mac will use multicast DNS queries (Bonjour) on the local network to get your Raspberry Pi’s IP address. You could also use the local IPv4 address of the Raspberry Pi. Or, if you have IPv6 configured, the globally-accessibly IPv6 address from anywhere in the world. [return]
  5. After cancelling backups a few times while Restic was running and attempting to run backups from multiple computers to the same repository at once, check reported that I needed to rebuild the repository index. I ran rebuild-index, and it appears to have worked. [return]
  6. In a way, this is emblematic of my entire experience of using Linux for the past two decades: Because everything is configurable and customizable, you can do some pretty cool stuff that. The flip side is that you can also shoot yourself in the foot. There are no training wheels, and if you don’t know as much as you think you do you can easily end up worse off than when you started. [return]
  7. I created a new account for him on the Raspberry Pi. And thanks to Restic’s built-in encryption, I literally have no way of seeing any of his data. [return]