restic: Encrypted Offsite Backup With Ransomware Protection for Your Homeserver

This article explains how to set up restic (with the resticprofile wrapper) for automated scheduled backups of your home server. The backups are protected from ransonmware through temporary immutability, which makes it much harder for attackers to delete your data. This post is part of my series on home automation, networking & self-hosting that shows how to install, configure, and run a home server with (dockerized or virtualized) services such as Home Assistant and ownCloud.

Requirements

Linux backup was a new topic for me, so I had to put in some research first. I started by compiling a list of requirements.

Must Have

• Client-side encryption
• Deduplication
• Metadata backup, e.g., symlinks & extended attributes (used by ownCloud)
• Off-site storage (preferably S3-compatible)
• Success/failure notifications (e.g., by email)
• Reliability & maturity
• Free software

Nice to Have

• Web UI

Storage Providers

As a second step, I looked for offsite storage providers.

S3-compatible

• Amazon AWS
• Backblaze B2

SSH/SFTP/SCP

• rsync.net
• BorgBase
  • Offers alerting if no backup has been performed for a configured interval.

Multi-Protocol

Hetzner has an interesting and inexpensive offering with their Storage Boxes:

• Access for restic via SFTP or Rclone (docs).
• SFTP may be slow.
• Rclone seems to be complex to set up (securely) (docs).

My Choice: Backblaze B2

I selected Backblaze B2 as my storage provider of choice. I wanted an S3-compatible provider because of their widespread availability and went for Backblaze because it’s inexpensive, easy to set up, and very fast.

Amazon S3, on the other hand, is a nightmare to set up and configure. Also, it’s much more expensive. Backblaze is the obvious choice for individuals and smaller organizations.

Backup Tools

Finally, I researched free Linux backup tools and was pleasantly surprised. There are both well-established products as well as newer modern developments.

Borg

Borg is a well-established, stable, and mature player that has been around for a long time.

• Off-site storage: remote repositories via SSH.
• Web UI: no
• Reliability & maturity: yes
• Email notifications: no

Borgmatic is a wrapper that provides:

• Configuration via YAML files
• Logging to syslog or file
• Monitoring options

restic

restic is a newer backup tool that has already become very popular.

• Off-site storage: remote repositories via SFTP, S3, and other protocols.
• Web UI: no
• Reliability & maturity:
  • Reliability: yes
  • Maturity: no. The project only guarantees no breaking changes starting with 1.0 (current version: 0.15.1).
• Email notifications: no

I found two restic wrappers that look promising:

• Autorestic
• resticprofile

Other Backup Tools

The following additional tools showed up in my research:

• kopia
  • Off-site storage: remote repositories via SFTP, S3, and other protocols.
  • Web UI: yes (in server mode), but doesn’t cover full functionality (e.g., actions are missing).
  • Maturity: no, still new
• Duplicati
  • Off-site storage: remote repositories via SSH, S3, and other protocols.
  • Web UI: yes
  • Maturity: yes

• knoxite
  • Off-site storage: remote repositories via SSH, S3, and other protocols.
  • Web UI: no
  • Maturity: no. No recent release; no binary release.

My Choice: restic/resticprofile

I selected restic with its resticprofile wrapper for the following reasons:

• Extremely fast.
• Ability to resume a backup:
  • Just run the backup command again; restic picks up where it left off.
  • Very useful when an SSH section gets disconnected.
• Support for Windows in addition to Linux.
• Scheduling via systemd as well as cron.
• Backup statistics and detailed logs are available.

Backblaze Storage Configuration

Ransomware Protection Through Temporary Immutability

restic doesn’t need delete permissions on S3-compatible access keys (source). This enables us to implement ransomware protection through temporary immutability.

The concept is simple enough: we create an application key without delete capabilities. If attackers get hold of our server and, therefore, of the key, they cannot wipe our existing backups from the Backblaze storage bucket.

restic, on the other hand, needs a way to “delete” files, of course. When it identifies a file that is no longer required, it overwrites it with a “hidden marker” (a non-destructive delete), creating a new file version. The original content is still available as an old version of the file.

To prevent the bucket from growing indefinitely, we configure lifecycle settings to keep prior versions for a certain number of days. I chose to err on the paranoid side and went for 365 days. If someone deletes my backups, I can restore them within one year with a tool like brestore.

Kudos to Benjamin Ritter, who described this simple but effective configuration in his blog post How to do ransomware resistant backups properly with Restic and Backblaze B2.

Create a Bucket

Create a Backblaze account and enable 2FA. Note that region selection is only possible during account setup. Choose between EU Central, US West, and US East.

Create a new bucket with the following properties:

• Private
• Encryption disabled
• Object lock disabled

Create an Application Key Without Delete Capabilities

Preparation: B2 Command-Line Tool (Windows)

• Download b2-windows.exe
• Open a command prompt (cmd.exe)
• Specify your master application key with temporary environment variables:

  set B2_APPLICATION_KEY_ID=YOUR_KEY_ID
  set B2_APPLICATION_KEY=YOUR_KEY
  

• Authorize the CLI tool:

  b2-windows.exe account authorize

  Note: as part of the authorization, the tool creates a file with cached credentials in your user profile (see below).

Create the Application Key

To create an application key with minimal capabilities for restic:

b2-windows.exe key create --bucket BUCKET_NAME restic-no-delete listBuckets,listFiles,readFiles,writeFiles 

The tool displays the key ID followed by the key itself. Copy both values.

Delete Cached Credentials

Delete the file %userprofile%\.b2_account_info that was created during authorization.

Restic Installation & Configuration

How to Backup Up Docker Container Data?

The easiest way to back up data from Docker containers is to stop the containers. This releases all file locks or handles, closes open connections, etc., so that even databases can be backed up by simple file copy operations.

Bind mounts simplify the task of cleanly organizing the container data, as you can see in the articles of this series. If you’re using Docker volumes instead, you probably need additional logic in your backup process to locate the volumes’ directories in the file system.

Dockerized or Natively Installed?

Before we can back up Docker container data, we need to stop all containers. The easiest way to accomplish that is to stop the Docker service altogether. Stopping Docker is not possible from within a container that needs to be kept running. I, therefore, installed restic natively on the host.

Installation

restic

Run the following commands to install restic and then use its self-update functionality to upgrade to the latest version:

apt install restic
restic self-update

resticprofile

Run the following commands to download and install resticprofile in /usr/local/bin:

curl -LO https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh
chmod +x install.sh
./install.sh -b /usr/local/bin

Updating resticprofile

Just like regular restic, resticprofile has the capability to self-update. Just run the following command:

resticprofile self-update

resticprofile Configuration File

We’ll store resticprofile’s configuration and password files on our encrypted ZFS dataset. Create the directory /rpool/encrypted/resticprofile. In this directory, create the configuration file profiles.yaml with the following content:

default:
  lock: "/tmp/resticprofile-profile-default.lock"
  force-inactive-lock: true
  initialize: true
  repository: "s3:BUCKET_ENDPOINT_URL/YOUR_BUCKET_NAME"     # Example: s3:s3.eu-central-003.backblazeb2.com/YOUR_BUCKET_NAME 
  password-file: "YOUR_BUCKET_NAME.key"
  status-file: "backup-status.json"
  run-before:
    - "systemctl stop docker.socket"
    - "systemctl stop docker"
  run-finally:
    - "systemctl start docker"

  env:
    AWS_ACCESS_KEY_ID: "YOUR_KEY_ID"
    AWS_SECRET_ACCESS_KEY: "YOUR_KEY"

  # Backup command
  backup:
    run-finally:
      # Extract lines that don't start with "unchanged " from the log and write them to "backup.log" in the current directory
      - 'grep --invert-match -E "^unchanged\\s" {{ tempFile "backup.log" }} > backup.log'
    one-file-system: true                                   # Don't leave the file system via mount points
    source:
      - "/etc/unbound/unbound.conf"
      - "/rpool/data/docker"
      - "/rpool/encrypted/docker"
      - "/rpool/encrypted/resticprofile"
    schedule: "04:00"
    schedule-permission: system
    schedule-lock-wait: 10m
    schedule-log: '{{ tempFile "backup.log" }}'             # Create log file "backup.log" in temporary directory and delete it when resticprofile is done
    verbose: 2                                              # Write details about each processed file to the log

  # Retention policy command
  forget:
    keep-daily: 7
    keep-weekly: 8
    keep-monthly: 12
    keep-yearly: 10
    prune: true
    schedule: "05:00"
    schedule-permission: system
    schedule-lock-wait: 1h

  # Verify command
  check:
    schedule: "06:00"
    schedule-permission: system
    schedule-lock-wait: 1h

Generate a random alphanumeric string and store it in the restic password file /rpool/encrypted/resticprofile/YOUR_BUCKET_NAME.key defined in the configuration:

tr -cd '[:alnum:]' < /dev/urandom | fold -w "64" | head -n 1 > /rpool/encrypted/resticprofile/YOUR_BUCKET_NAME.key

Important: keep a copy of the above key in a safe place; you’ll need it for decrypting your backups.

Backup Operations

Manual Backup

Run the following command to initialize the remote repository and perform the initial backup:

resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml backup

The output should look similar to the following:

2023/02/07 22:44:48 profile 'default': initializing repository (if not existing)
2023/02/07 22:44:49 profile 'default': starting 'backup'
repository 477f3d46 opened (repository version 2) successfully, password is correct
using parent snapshot e4f0d677

Files:         268 new,   164 changed, 178034 unmodified
Dirs:          669 new,  1067 changed, 521037 unmodified
Added to the repository: 368.899 MiB (226.759 MiB stored)

processed 178466 files, 394.210 GiB in 1:20
snapshot 60cdc5ee saved
2023/02/07 22:46:11 profile 'default': finished 'backup'

Note: The following error message is expected. As the setting name schedule-log implies, the backup.log is created only when resticprofile is running scheduled. See below for how to manually trigger a scheduled backup.

run-finally command 1/1 failed ('backup' on profile 'default'): %!w(*exec.ExitError=&{0xc000182390 []})

If extended-status is enabled in the YAML file, output from restic is not visible in the terminal, and the backup command’s output is reduced to what is emitted by resticprofile:

2023/02/07 22:44:48 profile 'default': initializing repository (if not existing)
2023/02/07 22:44:49 profile 'default': starting 'backup'
2023/02/07 22:46:11 profile 'default': finished 'backup'

Scheduling

The profiles.yaml configuration file already contains the schedule. The only thing left to do is instruct resticprofile to install it:

resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml schedule --all

The output should look similar to the following:

Analyzing backup schedule 1/1
=================================
  Original form: 04:00
Normalized form: *-*-* 04:00:00
    Next elapse: Wed 2023-02-08 04:00:00 CET
       (in UTC): Wed 2023-02-08 03:00:00 UTC
       From now: 4h 48min left

2023/02/07 23:11:04 writing /etc/systemd/system/[email protected]
2023/02/07 23:11:04 writing /etc/systemd/system/[email protected]
Created symlink /etc/systemd/system/timers.target.wants/[email protected] → /etc/systemd/system/[email protected].

Systemd timer status
=====================
● [email protected] - backup timer for profile default in /rpool/encrypted/resticprofile/profiles.yaml
     Loaded: loaded (/etc/systemd/system/[email protected]; enabled; vendor preset: enabled)
     Active: active (waiting) since Tue 2023-02-07 23:11:04 CET; 3ms ago
    Trigger: Wed 2023-02-08 04:00:00 CET; 4h 48min left
   Triggers: ● [email protected]

Feb 07 23:11:04 px1 systemd[1]: Started backup timer for profile default in /rpool/encrypted/resticprofile/profiles.yaml.
2023/02/07 23:11:04 scheduled job default/backup created

Status check

Run the following command to check the status of your scheduled resticprofile jobs:

resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml status

The output of the status command is similar to the schedule output shown above with the addition of the following helpful summary:

Timers summary
===============
NEXT                        LEFT          LAST PASSED UNIT                                       ACTIVATES
Wed 2023-02-08 04:00:00 CET 4h 48min left n/a  n/a    [email protected] [email protected]
Wed 2023-02-08 05:00:00 CET 5h 48min left n/a  n/a    [email protected] [email protected]
Wed 2023-02-08 06:00:00 CET 6h left       n/a  n/a    [email protected]  [email protected]

3 timers listed.

Manually Triggering a Scheduled Job

During testing, you may want to trigger scheduled jobs manually. Use the following command for that purpose:

systemctl start [email protected]

Configuration Updates

If you update the configuration file profiles.yaml, make sure to re-run the scheduling command (see above). resticprofile is clever enough to update the timers and service units without creating duplicates.

Snapshot Info

Listing Snapshots

To list the snapshots in the backup target:

resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml snapshots

The output looks similar to the following:

2023/06/03 12:48:46 profile 'default': initializing repository (if not existing)
2023/06/03 12:48:47 profile 'default': starting 'snapshots'
repository 12345678 opened (version 2, compression level auto)
ID        Time                 Host        Tags        Paths
-------------------------------------------------------------------------------------
11223344  2023-05-16 04:00:07  px1                     /rpool/data/docker
                                                       /rpool/encrypted/docker

22334455  2023-06-03 04:00:03  px1                     /rpool/data/docker
                                                       /rpool/encrypted/docker
                                                       /rpool/encrypted/resticprofile
-------------------------------------------------------------------------------------
2 snapshots

Snapshot Stats

To generate statistics for a given snapshot:

resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml stats 22334455

The output looks similar to the following:

2023/06/03 12:51:48 profile 'default': initializing repository (if not existing)
2023/06/03 12:51:49 profile 'default': starting 'stats'
repository 12345678 opened (version 2, compression level auto)
scanning...
Stats in restore-size mode:
     Snapshots processed:  1
        Total File Count:  793125
              Total Size:  393.812 GiB
2023/06/03 12:52:02 profile 'default': finished 'stats'

Mounting the Backup Repository

Restic’s capability to mount the remote backup repository into the local filesystem enables us to access the backup with any tools we want. To mount the backup repo run the following command:

mkdir /mnt/restic
resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml mount /mnt/restic

The output looks similar to the following:

2023/06/03 13:29:22 profile 'default': initializing repository (if not existing)
2023/06/03 13:29:23 profile 'default': starting 'mount'
repository 12345678 opened (version 2, compression level auto)
Now serving the repository at /mnt/restic/
Use another terminal or tool to browse the contents of this folder.
When finished, quit with Ctrl-c here or umount the mountpoint.

As the output states, the command keeps the repository mounted until you press Ctrl+C.

Inspecting the Contents of the Backup Repository

While the backup repository is mounted, open a second terminal to access and inspect the files, e.g.:

root@px1:~# la /mnt/restic/
total 0
dr-xr-xr-x 1 root root 0 Jun  3 13:29 hosts
dr-xr-xr-x 1 root root 0 Jun  3 13:29 ids
dr-xr-xr-x 1 root root 0 Jun  3 13:29 snapshots
dr-xr-xr-x 1 root root 0 Jun  3 13:29 tags
root@px1:~# la /mnt/restic/snapshots/
total 1
dr-xr-xr-x 2 root root  0 May 16 04:00 2023-05-16T04:00:07+02:00
dr-xr-xr-x 2 root root  0 Jun  3 04:00 2023-06-03T04:00:03+02:00
lrwxrwxrwx 1 root root 25 Jun  3 04:00 latest -> 2023-06-03T04:00:03+02:00
root@px1:~# la /mnt/restic/snapshots/latest/rpool/encrypted/resticprofile/
total 3
-rw-r--r-- 1 root root  541 May 16 06:01 backup-status.json
-rw-r--r-- 1 root root   65 Jan  9 23:25 helgeklein-wb28-restic-px1-1.key
-rw-r--r-- 1 root root 1147 Jun  3 00:53 profiles.yaml

Monitoring

Monitoring via Status File

The status-file line in the YAML file instructs resticprofile to generate a status file in JSON format with summary data on the last backup, forget, and check commands. The status file’s contents is similar to the following:

{
    "profiles":
    {
        "default":
        {
            "backup":
            {
                "success": true,
                "time": "2023-02-07T23:42:13.119348982+01:00",
                "error": "",
                "stderr": "",
                "duration": 42,
                "files_new": 819,
                "files_changed": 63,
                "files_unmodified": 178389,
                "dirs_new": 188,
                "dirs_changed": 199,
                "dirs_unmodified": 522574,
                "files_total": 179271,
                "bytes_added": 244728277,
                "bytes_total": 423386569555
            }
        }
    }
}

Note: Either extended-status or schedule-log needs to be configured for resticprofile to process restic’s output and generate meaningful statistics.

Monitoring via Backup Log

The following lines in the configuration file are responsible for the creation of a log file that lists detailed information about each file processed by restic:

  backup:
    schedule-log: '{{ tempFile "backup.log" }}'             # Create log file "backup.log" in temporary directory and delete it when resticprofile is done
    verbose: 2                                              # Write details about each processed file to the log
    run-finally:
      # Extract lines that don't start with "unchanged " from the log and write them to "backup.log" in the current directory
      - 'grep --invert-match -E "^unchanged\\s" {{ tempFile "backup.log" }} > backup.log'

Note: extended-status must not be enabled or the log file won’t even be created.

The mechanism above may look unnecessarily complex. There are two reasons it is the way it is: One, I wanted to be sure the log is deleted and recreated for every backup job. Two, the log becomes very large because it contains a line of text even for unchanged files, which I’m not interested in and wanted to remove. Please see the documentation on logs, variables, and templates for details.

Monitoring via Prometheus PROM File

Configure resticprofile to generate a Prometheus .prom file (docs) by adding the following line to your profiles.yaml:

default:
  prometheus-save-to-file: "/etc/node-exporter/resticprofile.prom"

Use the textfile collector of Prometheus’ node exporter to collect the .prom file as described in this article.

Monitoring via Grafana Dashboard

Please see my article resticprofile Backup Monitoring Grafana Dashboard for a ready-to-use dashboard implementation.

Resources

• Quickstart Guide for Restic and Backblaze B2 Cloud Storage
• Restic Backups with systemd and Prometheus exporter
• Restic Backup I — Simple and beautiful backups

Changelog

2024-05-14

• Config change to provide temporary immutability (ransomware protection).
• Config change from B2 to S3 to account for a warning not to use the B2 backend library in restic’s documentation.

2024-04-07

• Modified the path to the Prometheus PROM file.

2023-10-03

• Added a note explaining the run-finally error when manually running resticprofile.

2023-07-09

• Updated the configuration so that a backup log file is created and added the corresponding section Monitoring via Backup Log.
• Added the sections Manually Triggering a Scheduled Job and Configuration Updates.
• Added the section Monitoring via Grafana Dashboard.

2023-06-03

• Added the resticprofile config/data directory as a backup source.
• Added the section Snapshot Info.
• Added the section Mounting the Backup Repository.