restic: Encrypted Offsite Backup for Your Homeserver

This article explains how to set up restic (with the resticprofile wrapper) for automated scheduled backups of your home server. This post is part of my series on home automation that shows how to install, configure, and run a home server with (dockerized or virtualized) services such as Home Assistant and ownCloud.

Requirements

Linux backup was a new topic for me, so I had to put in some research first. I started by compiling a list of requirements.

Must Have

• Client-side encryption
• Deduplication
• Metadata backup, e.g., symlinks & extended attributes (used by ownCloud)
• Off-site storage (preferably S3-compatible)
• Success/failure notifications (e.g., by email)
• Reliability & maturity
• Free software

Nice to Have

• Web UI

Storage Providers

As a second step, I looked for offsite storage providers.

S3-compatible

• Amazon AWS
• Backblaze B2

SSH/SFTP/SCP

• rsync.net
• BorgBase
  • Offers alerting if no backup has been performed for a configured interval.

Multi-Protocol

Hetzner has an interesting and inexpensive offering with their Storage Boxes:

• Access for restic via SFTP or Rclone (docs).
• SFTP may be slow.
• Rclone seems to be complex to set up (securely) (docs).

My Choice: Backblaze B2

I selected Backblaze B2 as my storage provider of choice. I wanted an S3-compatible provider because of their widespread availability and went for Backblaze because it’s inexpensive, easy to set up, and very fast.

Amazon S3, on the other hand, is a nightmare to set up and configure. Also, it’s much more expensive. Backblaze is the obvious choice for individuals and smaller organizations.

Backup Tools

Finally, I researched free Linux backup tools and was pleasantly surprised. There are both well-established products as well as newer modern developments.

Borg

Borg is a well-established, stable, and mature player that has been around for a long time.

• Off-site storage: remote repositories via SSH.
• Web UI: no
• Reliability & maturity: yes
• Email notifications: no

Borgmatic is a wrapper that provides:

• Configuration via YAML files
• Logging to syslog or file
• Monitoring options

restic

restic is a newer backup tool that has already become very popular.

• Off-site storage: remote repositories via SFTP, S3, and other protocols.
• Web UI: no
• Reliability & maturity:
  • Reliability: yes
  • Maturity: no. The project only guarantees no breaking changes starting with 1.0 (current version: 0.15.1).
• Email notifications: no

I found two restic wrappers that look promising:

• Autorestic
• resticprofile

Other Backup Tools

The following additional tools showed up in my research:

• kopia
  • Off-site storage: remote repositories via SFTP, S3, and other protocols.
  • Web UI: yes (in server mode), but doesn’t cover full functionality (e.g., actions are missing).
  • Maturity: no, still new
• Duplicati
  • Off-site storage: remote repositories via SSH, S3, and other protocols.
  • Web UI: yes
  • Maturity: yes

• knoxite
  • Off-site storage: remote repositories via SSH, S3, and other protocols.
  • Web UI: no
  • Maturity: no. No recent release; no binary release.

My Choice: restic/resticprofile

I selected restic with its resticprofile wrapper for the following reasons:

• Extremely fast.
• Ability to resume a backup:
  • Just run the backup command again; restic picks up where it left off.
  • Very useful when an SSH section gets disconnected.
• Support for Windows in addition to Linux.
• Scheduling via systemd as well as cron.
• Backup statistics via a status file.

Backblaze Storage Configuration

Bucket & Application Key

Create a Backblaze account and enable 2FA. Note that region selection is only possible during account setup. Choose between EU Central, US West, and US East.

Create a new bucket with the following properties:

• Private
• Encryption disabled
• Object lock disabled

Create an application key with the following properties:

• Name of key: restic
• Allow access only to the bucket you just created
• Type of access: read and write

Restic Installation & Configuration

How to Backup Up Docker Container Data?

The easiest way to back up data from Docker containers is to stop the containers. This releases all file locks or handles, closes open connections, etc., so that even databases can be backed up by simple file copy operations.

Bind mounts simplify the task of cleanly organizing the container data, as you can see in the articles of this series. If you’re using Docker volumes instead, you probably need additional logic in your backup process to locate the volumes’ directories in the file system.

Dockerized or Natively Installed?

Before we can back up Docker container data, we need to stop all containers. The easiest way to accomplish that is to stop the Docker service altogether. Stopping Docker is not possible from within a container that needs to be kept running. I, therefore, installed restic natively on the host.

Installation

restic

Run the following commands to install restic and then use its self-update functionality to upgrade to the latest version:

apt install restic
restic self-update

resticprofile

Run the following commands to download and install resticprofile in /usr/local/bin:

curl -LO https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh
chmod +x install.sh
./install.sh -b /usr/local/bin

Updating resticprofile

Just like regular restic, resticprofile has the capability to self-update. Just run the following command:

resticprofile self-update

resticprofile Configuration File

We’ll store resticprofile’s configuration and password files on our encrypted ZFS dataset. Create the directory /rpool/encrypted/resticprofile. In this directory, create the configuration file profiles.yaml with the following content:

default:
  lock: "/tmp/resticprofile-profile-default.lock"
  force-inactive-lock: true		# Make it possible to restart canceled jobs
  initialize: true
  repository: "b2:YOUR_BUCKET_NAME"
  password-file: "YOUR_BUCKET_NAME.key"
  status-file: "backup-status.json"
  run-before:
    - "systemctl stop docker.socket"
    - "systemctl stop docker"
  run-finally:
    - "systemctl start docker"
  
  env:
    B2_ACCOUNT_ID: "YOUR_KEY_ID"
    B2_ACCOUNT_KEY: "YOUR_KEY"
    
  # Backup command
  backup:
    one-file-system: true			# Don't leave the file system via mount points
    source:
      - "/rpool/data/docker"
      - "/rpool/encrypted/docker"
    schedule: "04:00"
    schedule-permission: system
    schedule-lock-wait: 10m
    extended-status: true
    
  # Retention policy command
  forget:
    keep-daily: 7
    keep-weekly: 8
    keep-monthly: 12
    keep-yearly: 10
    prune: true
    schedule: "05:00"
    schedule-permission: system
    schedule-lock-wait: 1h
    
  # Verify command
  check:
    schedule: "06:00"
    schedule-permission: system
    schedule-lock-wait: 1h

Generate a random alphanumeric string and store it in the restic password file /rpool/encrypted/resticprofile/YOUR_BUCKET_NAME.key defined in the configuration:

tr -cd '[:alnum:]' < /dev/urandom | fold -w "64" | head -n 1 > /rpool/encrypted/resticprofile/YOUR_BUCKET_NAME.key

Important: keep a copy of the above key in a safe place; you’ll need it for decrypting your backups.

Backup Operations

Manual Backup

Run the following command to initialize the remote repository and perform the initial backup:

resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml backup

The output should look similar to the following:

2023/02/07 22:44:48 profile 'default': initializing repository (if not existing)
2023/02/07 22:44:49 profile 'default': starting 'backup'
repository 477f3d46 opened (repository version 2) successfully, password is correct
using parent snapshot e4f0d677

Files:         268 new,   164 changed, 178034 unmodified
Dirs:          669 new,  1067 changed, 521037 unmodified
Added to the repository: 368.899 MiB (226.759 MiB stored)

processed 178466 files, 394.210 GiB in 1:20
snapshot 60cdc5ee saved
2023/02/07 22:46:11 profile 'default': finished 'backup'

If extended-status is enabled in the YAML file, output from restic is not visible in the terminal, and the backup command’s output is reduced to what is emitted by resticprofile:

2023/02/07 22:44:48 profile 'default': initializing repository (if not existing)
2023/02/07 22:44:49 profile 'default': starting 'backup'
2023/02/07 22:46:11 profile 'default': finished 'backup'

Scheduling

The profiles.yaml configuration file already contains the schedule. The only thing left to do is instruct resticprofile to install it:

resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml schedule --all

The output should look similar to the following:

Analyzing backup schedule 1/1
=================================
  Original form: 04:00
Normalized form: *-*-* 04:00:00
    Next elapse: Wed 2023-02-08 04:00:00 CET
       (in UTC): Wed 2023-02-08 03:00:00 UTC
       From now: 4h 48min left

2023/02/07 23:11:04 writing /etc/systemd/system/[email protected]
2023/02/07 23:11:04 writing /etc/systemd/system/[email protected]
Created symlink /etc/systemd/system/timers.target.wants/[email protected] → /etc/systemd/system/[email protected]

Systemd timer status
=====================
● [email protected] - backup timer for profile default in /rpool/encrypted/resticprofile/profiles.yaml
     Loaded: loaded (/etc/systemd/system/[email protected]; enabled; vendor preset: enabled)
     Active: active (waiting) since Tue 2023-02-07 23:11:04 CET; 3ms ago
    Trigger: Wed 2023-02-08 04:00:00 CET; 4h 48min left
   Triggers: ● [email protected]

Feb 07 23:11:04 px1 systemd[1]: Started backup timer for profile default in /rpool/encrypted/resticprofile/profiles.yaml.
2023/02/07 23:11:04 scheduled job default/backup created

Status check

Run the following command to check the status of your scheduled resticprofile jobs:

resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml status

The output of the status command is similar to the schedule output shown above with the addition of the following helpful summary:

Timers summary
===============
NEXT                        LEFT          LAST PASSED UNIT                                       ACTIVATES
Wed 2023-02-08 04:00:00 CET 4h 48min left n/a  n/a    [email protected] [email protected]
Wed 2023-02-08 05:00:00 CET 5h 48min left n/a  n/a    [email protected] [email protected]
Wed 2023-02-08 06:00:00 CET 6h left       n/a  n/a    [email protected]  [email protected]

3 timers listed.

Monitoring

Monitoring via Status File

The status-file line in the YAML file instructs resticprofile to generate a status file in JSON format with summary data on the last backup, forget, and check commands. The status file’s contents is similar to the following:

{
    "profiles":
    {
        "default":
        {
            "backup":
            {
                "success": true,
                "time": "2023-02-07T23:42:13.119348982+01:00",
                "error": "",
                "stderr": "",
                "duration": 42,
                "files_new": 819,
                "files_changed": 63,
                "files_unmodified": 178389,
                "dirs_new": 188,
                "dirs_changed": 199,
                "dirs_unmodified": 522574,
                "files_total": 179271,
                "bytes_added": 244728277,
                "bytes_total": 423386569555
            }
        }
    }
}

Monitoring via Prometheus PROM File

Configure resticprofile to generate a Prometheus .prom file (docs) by adding the following line to your profiles.yaml:

default:
  prometheus-save-to-file: "resticprofile.prom"

Use the textfile collector of Prometheus’ node exporter to collect the .prom file. I’ll describe the Prometheus configuration in detail in a future article.

Resources

• Quickstart Guide for Restic and Backblaze B2 Cloud Storage
• Restic Backups with systemd and Prometheus exporter
• Restic Backup I — Simple and beautiful backups