by , last updated November 23, 2022, in

Installing Proxmox as Docker Host on Intel NUC Home Server

This is my first article in what is poised to become a series on installing, configuring, and running a home server with (dockerized or virtualized) services such as Home Assistant and OwnCloud.

Home Server Hardware

Selection Criteria

I was looking for the following when I selected my home server hardware:

  • Linux support
  • Small form factor
  • Fast CPU
    • Good single-thread performance
    • At least 4 cores
  • Low power consumption
  • 2.5 GbE

Hardware Model

I considered building my own, but good small form-factor (SFF) cases are rare, and the few interesting cases seem to be geared toward gamers. A home server does not need a powerful GPU, so I went with an Intel NUC instead. The Intel NUC 12 Pro Kit NUC12WSHi7 is a NUC variant with a taller case that can be fitted with a 2.5″ disk drive in addition to the M.2 SSD(s).

  • CPU: i7-1260P, mobile CPU, 28 W TDP (specs)
    • 12 cores (4 performance, 8 efficiency)
    • Cinebench R23 single-core: 1737
    • Cinebench R23 multi-core: 9743
  • RAM: 2 DIMMs, max. 64 GB
  • Disk options:
    • 1 x M.2 2280 NVMe SSD (PCIe x4)
    • 1 x M.2 2242 SATA SSD (PCIe x1)
    • 1 x 2.5″ SATA disk
  • Supported operating systems: Windows 10/11, Red Hat Linux, Ubuntu 20.04 LTS

Home Server OS: Why Proxmox?

A home server is, first and foremost, a workhorse, just like any other server. Unlike a server in a company’s data center, though, a home server is part of a long-lived structure, a building. That is why a home server needs an operating system that is long-term stable and can be run without frequent “hand-holding” (aka administration). Also, a non-commercial (free) license is more than welcome.

Given these requirements, Linux is the OS of choice. Proxmox, with its good ZFS support and functional web UI, is the obvious distribution.

Installing Proxmox

Preparation

Update the NUC’s BIOS:

  • Copy the BIOS .CAP file to a USB drive
  • Press F7 during boot
  • Select the .CAP file and start the update

Disable Secure Boot in the BIOS (docs):

  • Press F2 during boot to enter the BIOS
  • Go to Advanced > Boot > Secure Boot
  • Disable Secure Boot

Proxmox Installation

In place of a screen-by-screen replay of the installation process, I’ll simply list the relevant settings:

  • File system: ZFS
  • Swap partition (1)
    • The installer does not create a swap partition
    • Locating the swap partition on ZFS is a bad idea because ZFS needs lots of RAM
    • Instead, reserve 8 GB in the advanced options of Proxmox’s installer
      • Select Target > Options
      • Filesystem: zfs (RAID0)
      • Open Advanced Options
      • Reduce hdsize by 8
  • Network configuration
    • IP address
      • DHCP works, but is not officially supported (config).
      • The official DNS domain for home networks is home.arpa (RFC8375).
      • I’m using a public DNS domain instead because I want to use Let’s encrypt certificates verified through the DNS challenge (details in a future article).
    • To change the IP address, edit these files:
      • /etc/network/interfaces
      • /etc/hosts
    • To change the hostname or DNS domain, edit these files:
      • /etc/hosts
      • /etc/hostname

Initial Configuration

Update Proxmox

If you’re using the free version of Proxmox VE (i.e., you don’t have a subscription), update repositories as follows:

  • Web UI > INSTANCE > Updates > Repositories > Add
    • Add the No-Subscription repository
  • Disable the enterprise repository (because it requires a subscription)

Upgrade Proxmox:

  • Click Refresh to run apt update
  • Click Upgrade to run apt dist-upgrade

Disk Setup

Swap Partition

Create & enable the swap partition (docs):

  • Show swap config: swapon -s
    • There should be no output = no swap partition
  • Run cfdisk to create a partition in the 8 GB of free space
    • This creates /dev/sda4.
  • Run mkswap to get the UUID: mkswap /dev/sda4
  • Enable swapping: swapon -U <UUID>
  • Add the following line to /etc/fstab: UUID=<UUID> none swap sw 0 0
SSD Trim
  • Check if trim is enabled: zpool get autotrim rpool
  • Enable trim if disabled: zpool set autotrim=on rpool

ZFS Storage Setup

Change Default ZFS Storage Name

The installer created a ZFS dataset with the name local-zfs. Rename it to zfs-data by editing /etc/pve/storage.cfg.

Create Encrypted ZFS Dataset

Set up an encrypted dataset:

zfs create -o encryption=on -o keyformat=passphrase rpool/encrypted

Create a new storage location in the UI: Web UI > Datacenter > Storage > Add:

  • ID: zfs-data-encrypted
  • ZFS Pool: rpool/encrypted
  • Content: Disk image, Container
  • Thin provision: checked

Note: when you reboot, it will show your VMs but won’t be able to access or start them until you access the command line for that Proxmox host and run: zfs mount -l rpool/encrypted.

Create ZFS Dataset for Images
  • Create a ZFS dataset for VM/container images: zfs create rpool/vmdata
  • Enable compression: zfs set compression=on rpool/vmdata
  • Create a new storage location in the UI: Web UI > Datacenter > Storage > Add
    • ID: zfs-images
    • ZFS Pool: rpool/vmdata
    • Content: Disk image, Container
    • Thin provision: checked

Network Setup

Network Bridge with NAT

The Proxmox setup should have created the bridged network vmbr0 where VMs/containers have direct access to the network. We’ll add a second internal network with NAT for VMs/containers that don’t need to be reachable from outside.

Add the following to /etc/network/interfaces:

auto vmbr1
iface vmbr1 inet static
   address        10.10.10.1/24
   bridge-ports   none
   bridge-stp     off
   bridge-fd      0

   post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
   post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE
   post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o vmbr0 -j MASQUERADE

Reboot the Proxmox host. You can verify the interface status with ip a. Note, though, that the newly added NAT interface vmbr1 will show as NO-CARRIER until you connect a VM or container to it.

Two-Factor Authentication (2FA)

  • Enable TOTP for the current (root) user: Web UI > Datacenter > Permissions > Two Factor > Add > TOTP.
  • Generate recovery keys for the root user: Web UI > Datacenter > Permissions > Two Factor > Add > Recovery Keys.

SSH access

Create a key pair on your PC and upload it to your user’s SSH key file as described here. Once you’ve verified public key authentication, disable password authentication as described in the linked article.

Installing Docker

Docker in an Unprivileged LXC Container?

While it’s possible to run Docker in an unprivileged LXC container, it introduces another abstraction layer, increasing complexity. It’s also too unstable to be used in production. Since I’m looking for low maintenance and long-term stability, I decided to install Docker natively on the Proxmox host.

By the way, I decided against rootless Podman for similar reasons. The official list of shortcomings is simply to long.

Install Docker on Proxmox

We’re installing Docker along with Docker Compose from the official Docker repository according to the docs:

Update:

apt update

Install the required packages to access the Docker repository via HTTPS:

apt install ca-certificates curl gnupg lsb-release

Add Docker’s GPG key:

mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg

Add Docker’s stable repository:

echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

Update and install Docker:

apt update
apt install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Verify that Docker is installed correctly by running an image that prints a message and exits:

systemctl status docker
docker run hello-world

Configure Docker

Log Rotation

By default, log rotation is disabled (docs). To enable log rotation, create a Docker config file /etc/docker/daemon.json with the following content:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Restart the Docker daemon (service docker restart). To verify, run docker info.

Docker Data Directory

On the Proxmox host, create a Docker data directory: mkdir /rpool/data/docker.

Troubleshooting Docker

Inspect Container Logs

Docker captures stdout and stderr output from within containers and writes it to log files. To inspect, use docker compose logs (docker logs works, too), e.g., like this:

docker compose logs <container> --tail 30 --timestamps

Resources

Previous Article DNS Exfiltration & Tunneling: How it Works & DNSteal Demo Setup