restic: Encrypted Offsite Backup for Your Homeserver
This article explains how to set up restic (with the resticprofile wrapper) for automated scheduled backups of your home server. This post is part of my series on home automation that shows how to install, configure, and run a home server with (dockerized or virtualized) services such as Home Assistant and ownCloud.
Requirements
Linux backup was a new topic for me, so I had to put in some research first. I started by compiling a list of requirements.
Must Have
- Client-side encryption
- Deduplication
- Metadata backup, e.g., symlinks & extended attributes (used by ownCloud)
- Off-site storage (preferably S3-compatible)
- Success/failure notifications (e.g., by email)
- Reliability & maturity
- Free software
Nice to Have
- Web UI
Storage Providers
As a second step, I looked for offsite storage providers.
S3-compatible
- Amazon AWS
- Backblaze B2
SSH/SFTP/SCP
Multi-Protocol
Hetzner has an interesting and inexpensive offering with their Storage Boxes:
- Access for restic via SFTP or Rclone (docs).
- SFTP may be slow.
- Rclone seems to be complex to set up (securely) (docs).
My Choice: Backblaze B2
I selected Backblaze B2 as my storage provider of choice. I wanted an S3-compatible provider because of their widespread availability and went for Backblaze because it’s inexpensive, easy to set up, and very fast.
Amazon S3, on the other hand, is a nightmare to set up and configure. Also, it’s much more expensive. Backblaze is the obvious choice for individuals and smaller organizations.
Backup Tools
Finally, I researched free Linux backup tools and was pleasantly surprised. There are both well-established products as well as newer modern developments.
Borg
Borg is a well-established, stable, and mature player that has been around for a long time.
- Off-site storage: remote repositories via SSH.
- Web UI: no
- Reliability & maturity: yes
- Email notifications: no
Borgmatic is a wrapper that provides:
- Configuration via YAML files
- Logging to syslog or file
- Monitoring options
restic
restic is a newer backup tool that has already become very popular.
- Off-site storage: remote repositories via SFTP, S3, and other protocols.
- Web UI: no
- Reliability & maturity:
- Reliability: yes
- Maturity: no. The project only guarantees no breaking changes starting with 1.0 (current version: 0.15.1).
- Email notifications: no
I found two restic wrappers that look promising:
Other Backup Tools
The following additional tools showed up in my research:
- kopia
- Off-site storage: remote repositories via SFTP, S3, and other protocols.
- Web UI: yes (in server mode), but doesn’t cover full functionality (e.g., actions are missing).
- Maturity: no, still new
- Duplicati
- Off-site storage: remote repositories via SSH, S3, and other protocols.
- Web UI: yes
- Maturity: yes
- knoxite
- Off-site storage: remote repositories via SSH, S3, and other protocols.
- Web UI: no
- Maturity: no. No recent release; no binary release.
My Choice: restic/resticprofile
I selected restic with its resticprofile wrapper for the following reasons:
- Extremely fast.
- Ability to resume a backup:
- Just run the backup command again; restic picks up where it left off.
- Very useful when an SSH section gets disconnected.
- Support for Windows in addition to Linux.
- Scheduling via systemd as well as cron.
- Backup statistics via a status file.
Backblaze Storage Configuration
Bucket & Application Key
Create a Backblaze account and enable 2FA. Note that region selection is only possible during account setup. Choose between EU Central
, US West
, and US East
.
Create a new bucket with the following properties:
- Private
- Encryption disabled
- Object lock disabled
Create an application key with the following properties:
- Name of key:
restic
- Allow access only to the bucket you just created
- Type of access: read and write
Restic Installation & Configuration
How to Backup Up Docker Container Data?
The easiest way to back up data from Docker containers is to stop the containers. This releases all file locks or handles, closes open connections, etc., so that even databases can be backed up by simple file copy operations.
Bind mounts simplify the task of cleanly organizing the container data, as you can see in the articles of this series. If you’re using Docker volumes instead, you probably need additional logic in your backup process to locate the volumes’ directories in the file system.
Dockerized or Natively Installed?
Before we can back up Docker container data, we need to stop all containers. The easiest way to accomplish that is to stop the Docker service altogether. Stopping Docker is not possible from within a container that needs to be kept running. I, therefore, installed restic natively on the host.
Installation
restic
Run the following commands to install restic and then use its self-update functionality to upgrade to the latest version:
apt install restic
restic self-update
resticprofile
Run the following commands to download and install resticprofile in /usr/local/bin
:
curl -LO https://raw.githubusercontent.com/creativeprojects/resticprofile/master/install.sh
chmod +x install.sh
./install.sh -b /usr/local/bin
Updating resticprofile
Just like regular restic, resticprofile has the capability to self-update. Just run the following command:
resticprofile self-update
resticprofile Configuration File
We’ll store resticprofile’s configuration and password files on our encrypted ZFS dataset. Create the directory /rpool/encrypted/resticprofile
. In this directory, create the configuration file profiles.yaml
with the following content:
default:
lock: "/tmp/resticprofile-profile-default.lock"
force-inactive-lock: true # Make it possible to restart canceled jobs
initialize: true
repository: "b2:YOUR_BUCKET_NAME"
password-file: "YOUR_BUCKET_NAME.key"
status-file: "backup-status.json"
run-before:
- "systemctl stop docker.socket"
- "systemctl stop docker"
run-finally:
- "systemctl start docker"
env:
B2_ACCOUNT_ID: "YOUR_KEY_ID"
B2_ACCOUNT_KEY: "YOUR_KEY"
# Backup command
backup:
one-file-system: true # Don't leave the file system via mount points
source:
- "/rpool/data/docker"
- "/rpool/encrypted/docker"
schedule: "04:00"
schedule-permission: system
schedule-lock-wait: 10m
extended-status: true
# Retention policy command
forget:
keep-daily: 7
keep-weekly: 8
keep-monthly: 12
keep-yearly: 10
prune: true
schedule: "05:00"
schedule-permission: system
schedule-lock-wait: 1h
# Verify command
check:
schedule: "06:00"
schedule-permission: system
schedule-lock-wait: 1h
Generate a random alphanumeric string and store it in the restic password file /rpool/encrypted/resticprofile/YOUR_BUCKET_NAME.key
defined in the configuration:
tr -cd '[:alnum:]' < /dev/urandom | fold -w "64" | head -n 1 > /rpool/encrypted/resticprofile/YOUR_BUCKET_NAME.key
Important: keep a copy of the above key in a safe place; you’ll need it for decrypting your backups.
Backup Operations
Manual Backup
Run the following command to initialize the remote repository and perform the initial backup:
resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml backup
The output should look similar to the following:
2023/02/07 22:44:48 profile 'default': initializing repository (if not existing)
2023/02/07 22:44:49 profile 'default': starting 'backup'
repository 477f3d46 opened (repository version 2) successfully, password is correct
using parent snapshot e4f0d677
Files: 268 new, 164 changed, 178034 unmodified
Dirs: 669 new, 1067 changed, 521037 unmodified
Added to the repository: 368.899 MiB (226.759 MiB stored)
processed 178466 files, 394.210 GiB in 1:20
snapshot 60cdc5ee saved
2023/02/07 22:46:11 profile 'default': finished 'backup'
If extended-status
is enabled in the YAML file, output from restic is not visible in the terminal, and the backup command’s output is reduced to what is emitted by resticprofile:
2023/02/07 22:44:48 profile 'default': initializing repository (if not existing)
2023/02/07 22:44:49 profile 'default': starting 'backup'
2023/02/07 22:46:11 profile 'default': finished 'backup'
Scheduling
The profiles.yaml
configuration file already contains the schedule. The only thing left to do is instruct resticprofile to install it:
resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml schedule --all
The output should look similar to the following:
Analyzing backup schedule 1/1
=================================
Original form: 04:00
Normalized form: *-*-* 04:00:00
Next elapse: Wed 2023-02-08 04:00:00 CET
(in UTC): Wed 2023-02-08 03:00:00 UTC
From now: 4h 48min left
2023/02/07 23:11:04 writing /etc/systemd/system/[email protected]
2023/02/07 23:11:04 writing /etc/systemd/system/[email protected]
Created symlink /etc/systemd/system/timers.target.wants/[email protected] → /etc/systemd/system/[email protected]
Systemd timer status
=====================
● [email protected] - backup timer for profile default in /rpool/encrypted/resticprofile/profiles.yaml
Loaded: loaded (/etc/systemd/system/[email protected]; enabled; vendor preset: enabled)
Active: active (waiting) since Tue 2023-02-07 23:11:04 CET; 3ms ago
Trigger: Wed 2023-02-08 04:00:00 CET; 4h 48min left
Triggers: ● [email protected]
Feb 07 23:11:04 px1 systemd[1]: Started backup timer for profile default in /rpool/encrypted/resticprofile/profiles.yaml.
2023/02/07 23:11:04 scheduled job default/backup created
Status check
Run the following command to check the status of your scheduled resticprofile jobs:
resticprofile -c /rpool/encrypted/resticprofile/profiles.yaml status
The output of the status
command is similar to the schedule
output shown above with the addition of the following helpful summary:
Timers summary
===============
NEXT LEFT LAST PASSED UNIT ACTIVATES
Wed 2023-02-08 04:00:00 CET 4h 48min left n/a n/a [email protected] [email protected]
Wed 2023-02-08 05:00:00 CET 5h 48min left n/a n/a [email protected] [email protected]
Wed 2023-02-08 06:00:00 CET 6h left n/a n/a [email protected] [email protected]
3 timers listed.
Monitoring
Monitoring via Status File
The status-file
line in the YAML file instructs resticprofile to generate a status file in JSON format with summary data on the last backup
, forget
, and check
commands. The status file’s contents is similar to the following:
{
"profiles":
{
"default":
{
"backup":
{
"success": true,
"time": "2023-02-07T23:42:13.119348982+01:00",
"error": "",
"stderr": "",
"duration": 42,
"files_new": 819,
"files_changed": 63,
"files_unmodified": 178389,
"dirs_new": 188,
"dirs_changed": 199,
"dirs_unmodified": 522574,
"files_total": 179271,
"bytes_added": 244728277,
"bytes_total": 423386569555
}
}
}
}
Monitoring via Prometheus PROM File
Configure resticprofile to generate a Prometheus .prom
file (docs) by adding the following line to your profiles.yaml
:
default:
prometheus-save-to-file: "resticprofile.prom"
Use the textfile collector of Prometheus’ node exporter to collect the .prom
file. I’ll describe the Prometheus configuration in detail in a future article.
1 Comment
Thank you for writing this! It was extremely helpful to me to follow and set up my own Restic backup solution. I’m looking forward to the Prometheus configuration!