Automatic HTTPS Certificates for Services on Internal Home Network

This article explains how to set up automatic HTTPS certificates via Let’s Encrypt for services on your internal home network without opening a port on your firewall. It’s part of my series on home automation that shows how to install, configure, and run a home server with (dockerized or virtualized) services such as Home Assistant and ownCloud.

Requirements

Today, encryption of data in transit via HTTPS is considered mandatory and is often easy to implement via the excellent free service Let’s Encrypt. However, Let’s Encrypt typically requires that the service in question be reachable from the internet for domain verification purposes. Exposing internal services to the internet is often not desirable, however, as it makes those same services attackable. I was therefore looking for a way to obtain Let’s Encrypt certificates without creating a security risk by opening firewall ports. In a nutshell:

• Internal services on a home network are to be made accessible via HTTPS without certificate warnings.
• No access to internal services from the internet.
  • No open ports.
  • No cloud-based tunneling service.

DNS Challenge + Caddy to the Rescue

Luckily, there is an alternative to the Let’s Encrypt HTTP challenge: DNS. There’s also an excellent free reverse proxy that happily handles everything related to HTTPS, including the Let’s Encrypt DNS challenge: Caddy. This is how my setup works:

• We use a subdomain of a public DNS domain for the hostnames of our services.
  • For security reasons, the subdomain should be used exclusively for the purposes described in this article, not for email (see below).
• HTTPS certificates are handled by a Caddy instance that acts as a reverse proxy.
• Caddy is configured to auto-manage Let’s Encrypt certificates via the DNS challenge, which uses TXT records for verification.

Let’s Encrypt DNS Challenge Explained

Here’s what happens when a certificate is requested via the Let’s Encrypt DNS challenge:

• The Let’s Encrypt client creates a special _acme-challenge DNS TXT record.
• Let’s Encrypt queries DNS for that record. If the record’s data is in order, Let’s Encrypt issues the certificate.
• The Let’s Encrypt client deletes the _acme-challenge DNS TXT record as it’s not needed any more.

Let’s Encrypt DNS Challenge & DNS Zone Security

• API write access to the DNS record _acme-challenge is required for automatic renewal.
• Problem: most DNS providers don’t have granular access control. API tokens are often valid for the entire zone or even account (Cloudflare). This would allow modification of existing MX records (redirecting your email).
• DNS alias mode could be a solution (but isn’t).
  • In DNS alias mode, we’d set up a second DNS zone used exclusively for ACME (Let’s Encrypt) validation. API access is only required for this validation domain.
  • We’d set up a CNAME record to point from the main domain to the validation domain.
  • Unfortunately, this doesn’t work with Caddy’s Cloudflare DNS module.
• Instead, we use a DNS domain exclusively for the purposes described in this article (i.e., ACME challenges and, optionally, name resolution).

Preparation

Cloudflare API Token

I’m using a free Cloudflare account to manage the DNS domain for the hostnames of my services. Alternatively, you can use any DNS provider that’s supported by Caddy (search the list of modules for dns.providers).

In your Cloudflare account, create an API token with the following properties:

• Required permissions:
  • Zone – zone – read
  • Zone – DNS – edit
• Scope: Include all zones
  • Unfortunately, access to all zones is required. If you have multiple domains in the account, this grants edit permissions to all of them. Use a dedicated Cloudflare account with a single DNS zone.
  • If you don’t grant access to all zones, you get errors like the following from Caddy:
    "logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"host.home.yourdomain.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[host.home.yourdomain.com] solving challenges: presenting for challenge: adding temporary record for zone \"yourdomain.com.\": expected 1 zone, got 0 for yourdomain.com."

DNS & Name Resolution

Host Name Resolution

Let’s Encrypt certificates are for hostnames only, IP addresses are not included (which would technically be possible, it’s just not covered by Let’s Encrypt). It is, therefore, not possible to access the services via HTTPS by IP address, e.g. https://192.168.0.4. Instead, we access the services by fully-qualified domain name (FQDN) and need a way to resolve those names into IP addresses. For this example, we need to add one A record for the whoami service we’re testing with (see below):

whoami.home.yourdomain.com 192.168.0.4   # replace with your Docker host's IP address

Variant A: Your Own Internal DNS Server

This is the preferred name resolution method if you have your own internal DNS server. See my article on Unbound for detailed information on how to set up and configure Unbound as your own DNS resolver.

Variant B: Public DNS Domain

If you have not (yet) set up your own DNS server, you can use your public DNS domain instead. If you are using Cloudflare, set Proxy status to DNS only. Once you’ve saved, the record’s Proxy status shows DNS only - reserved IP. That’s OK.

Try to nslookup your new A record. You might get the following error:

$ nslookup whoami.home.yourdomain.com
Server:  fritz.box
Address:  fd00::ca0e:14ff:fe0a:6233

*** No internal type for both IPv4 and IPv6 Addresses (A+AAAA) records available for whoami.home.yourdomain.com

If that is the case, your home router or your DNS provider probably has DNS rebind protection enabled. Depending on the device or service, it may be easy to add exclusions for your public domain:

• Fritz!Box routers: KB article describing how to add exceptions
• NextDNS: add your domain to the allowlist

Caddy Container on Proxmox VE

Run through the preparatory steps to set up Docker on the Proxmox host (see this article for details).

Preparation: Increase UDP Buffer Sizes for QUIC

The QUIC protocol (implemented by Caddy) requires larger buffers than are normally available in Linux (source). Add the following to /etc/sysctl.conf:

net.core.rmem_max=2500000
net.core.wmem_max=2500000

Reboot and check the values with the following commands:

sysctl net.core.rmem_max
sysctl net.core.wmem_max

Dockerized Caddy Directory Structure

This is what the directory structure will look like when we’re done:

/rpool/
 └── data/
     └── docker/
         └── caddy/
             ├── config/
             ├── data/
             ├── dockerfile-dns/
                 └── Dockerfile
             ├── container-vars.env
             ├── Caddyfile
             └── docker-compose.yml

Create the caddy directory. The subdirectories config and data are created by docker compose on the first run.

Customized Caddy Docker Image

We need a custom Caddy Docker image that includes the Cloudflare DNS plugin, which is required for the Let’s Encrypt DNS challenge.

Create the file dockerfile-dns/Dockerfile with the following content:

ARG VERSION=2

FROM caddy:${VERSION}-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare

FROM caddy:${VERSION}

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

Note the second FROM instruction – this produces a much smaller image by simply overlaying the newly-built binary on top of the regular caddy image (docs).

Caddy container-vars.env File

Everything that is specific to your deployment goes into the container-vars.env file. This includes domain names, IP addresses, API keys, email addresses, and so on.

Create container-vars.env with the following content:

MY_DOMAIN=home.yourdomain.com               # replace with your domain
MY_HOST_IP=192.168.0.4                      # replace with your Docker host's IP address
CLOUDFLARE_API_TOKEN=<YOUR_TOKEN>           # add your token

Caddy Docker Compose File

Create docker-compose.yml with the following content:

services:

  caddy:
    build: ./dockerfile-dns
    container_name: caddy
    hostname: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp"
    networks:
      - caddynet
    env_file:
      - container-vars.env
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./data:/data
      - ./config:/config

  whoami:
    image: "containous/whoami"
    container_name: "whoami"
    hostname: "whoami"
    networks:
      - caddynet

networks:

  caddynet:
    attachable: true
    driver: bridge

The file and directories in the section volumes are bind-mounted from the Docker host. Their content persists when the container is destroyed. Caddyfile is mounted read-only (:ro).

The whoami service is created strictly for testing purposes. You can remove it once things are working as expected.

Caddyfile for the whoami Container

Create Caddy’s configuration file Caddyfile in the same directory as the .yml file and paste the following content:

whoami.{$MY_DOMAIN} {
	reverse_proxy whoami:80
	tls {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	}
}

Start the Containers

Navigate into the directory with the .yml file and run:

docker compose up -d

Inspect the container logs for errors with the command docker compose logs --tail 30 --timestamps.

Test

Open https://whoami.home.yourdomain.com in your browser. It should display without certificate warnings or errors.

Proxmox Let’s Encrypt Certificate

Proxmox is accessible via HTTPS exclusively but comes, understandably, only with a self-signed certificate. Proxmox’s built-in support for Let’s Encrypt does not include the DNS challenge, but we now have everything in place to use our Caddy container to proxy access to the host’s web interface, too.

Proxmox Caddyfile

Add the following to Caddyfile:

proxmox.{$MY_DOMAIN} {
	reverse_proxy {$MY_HOST_IP}:8006 {
      transport http {
         tls_insecure_skip_verify
      }
   }
	tls {
		dns cloudflare {env.CLOUDFLARE_API_TOKEN}
	}
}

DNS A Record

Add the following A record to your DNS domain:

proxmox.home.yourdomain.com 192.168.0.4     # replace with your Docker host's IP address

Reload Caddy’s Configuration

Instruct Caddy to reload its configuration by running:

docker exec -w /etc/caddy caddy caddy reload

You should now be able to access the Proxmox web interface at https://proxmox.home.yourdomain.com without getting a certificate warning from your browser.

Operations & Maintenance

Upgrading the Caddy Container

Run the following commands to upgrade the Caddy container:

cd /rpool/data/docker/caddy/
docker compose build --pull
docker compose up -d

After the upgrade, I’d advise to check the log for errors:

docker compose logs --tail 100 --timestamps

Removing Obsolete Docker Images

Run the following commands to clean up and removed Docker images that are not used anymore after the container upgrades:

docker system prune

Resources

• Caddy
  • How to use DNS provider modules in Caddy 2
  • Caddy v2 Reverse Proxy with Docker & Cloudflare DNS
• Let’s Encrypt (ACME)
  • A Technical Deep Dive: Securing the Automation of ACME DNS Challenge Validation

Changelog

2024-04-14

• Removed the version from docker-compose.yml; a warning mentions that it’s obsolete.

2023-10-14

• Added the section Preparation: Increase UDP Buffer Sizes for QUIC

2023-06-05

• Updated the DNS & Name Resolution section, adding the internal DNS server as the preferred name resolution method.

2022-12-17

• Replace the Caddy container’s file .env with container-vars.env for reasons of consistency with other containers in this series.

2022-12-04

• Renamed the Docker network caddy to caddynet.