Hosting Lemmy with Traefik - Leddit Social

I’ve seen a couple of instances around asking if it’s possible to self host Lemmy fronted by Traefik, figured I can share my test setup with the community. This configuration is still in testing phase but seems to work in a federated configuration. ## The Design To run in an elastic and highly available way (where possible with my current hardware) I’ve opted to use Docker Swarm as it’s one of the easiest ways to get started with multi-node deployment. There are a few limitations with Swarm and I’m planning on moving to a K3s cluster in the future. For added protection I’ve chosen to front my services with Cloudflare for the time being, there’s a definite tradeoff of privacy vs security here and since I’m not able to afford traffic scrubbing center and deal with any ISP issues for now it’ll be used. ## The Setup A few prerequisites are needed for this type of deployment although I think it should be fairly easy to adapt it to a single node setup. * Docker Swarm - 2x Manger nodes 2x Worker nodes * NFS - TrueNAS, used to store LE certs and pictrs blobs * Cloudflare’d domain - optional but makes life much easier * PostgreSQL Database - A dedicated postgresql VM for persistent database ## The Config I’ll skip some of the basic setup configurations since there are plenty of guides out there and focus on the docker compose files for the cloudflare, traefik, and lemmy stacks. Starting with the simplest, you can follow any guide to create a new tunnel to use for this setup, the compose file should look something like this: cloudflare/docker-compose.yml yaml version: "3.8" services: cloudflare: image: cloudflare/cloudflared:2023.5.1-amd64 # always hardcode your versions to avoid unexpected changes deploy: mode: global # in swarm we will deploy one instance per manager node (more nodes more redudnacy/LB) placement: constraints: - node.role == manager networks: - bridge # a default created network allowing connectivity to the rest of the network (and internet to connect upstream) - traefik_dmz # an overlay internal private network for LB'd services command: - tunnel - --no-autoupdate - run - --token=*** # replace with the token issued by cloudflare networks: bridge: external: true traefik_dmz: external: true > important: Once you’ve created your tunnal add a public host pointing your-lemmy.domain to https://traefik and under TLS enable the No TLS Verify option. > Since both cloudflare and traefik services are on a common network cloudflare can access the internal service VIP by calling traefik (matching the service name) traefik/docker-compose.yml yaml version: '3.8' services: traefik: image: traefik:v2.10 environment: - "CF_DNS_API_TOKEN=***" # your cloudflare API token with zone read and DNS edit permissions ports: - target: 80 published: 80 protocol: tcp mode: host - target: 443 published: 443 protocol: tcp mode: host - target: 8082 published: 8082 protocol: tcp mode: host deploy: mode: global placement: constraints: - node.role == manager labels: - traefik.enable=true - traefik.docker.network=traefik_dmz - traefik.http.routers.traefik-dashboard.rule=HostRegexp(`{subhost:your-hostname-pattern[\d]+}.homelab`) # To access the dashboard you can use your hosts' FQDN pattern and access https://your-hostname-pattern1.homelab - traefik.http.routers.traefik-dashboard.entrypoints=https - traefik.http.routers.traefik-dashboard.tls=true - traefik.http.routers.traefik-dashboard.service=api@internal - traefik.http.services.traefik.loadbalancer.server.port=8080 command: - "--log=true" # System logs good for debugging - "--log.level=WARN" - "--accesslog=false" # Access logs good for debugging but so noisy - "--accesslog.format=json" - "--accesslog.fields.defaultmode=keep" - "--accesslog.fields.headers.defaultmode=keep" - "--accesslog.fields.headers.names.Authorization=drop" - "--api=true" # enable for dashboard access - "--api.dashboard=true" - "--ping=true" # enable for external LB health checks - "--ping.entrypoint=ping" - "--serverstransport.insecureskipverify=true" - "--global.checknewversion=false" - "--global.sendanonymoususage=false" - "--entrypoints.http=true" - "--entrypoints.http.address=:80" - "--entrypoints.http.http.redirections.entrypoint.to=https" - "--entrypoints.http.http.redirections.entrypoint.scheme=https" - "--entrypoints.https=true" - "--entrypoints.https.address=:443" - "--entrypoints.https.forwardedheaders.insecure=true" - "--entrypoints.https.forwardedheaders.trustedips=10.0.0.0/8" # limit the x-forwarded-for header trust to your external LB - "--entrypoints.ping=true" - "--entrypoints.ping.address=:8082" - "--providers.docker=true" # This will allow us to auto detect configured services and forward traffic - "--providers.docker.exposedbydefault=false" - "--providers.docker.swarmmode=true" - "--certificatesresolvers.cloudflare.acme.dnschallenge=true" # LetsEncrypt using cloudflare DNS challenge - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare" - "--certificatesresolvers.cloudflare.acme.caserver=https://acme-v02.api.letsencrypt.org/directory" - "[email protected]" - "--certificatesresolvers.cloudflare.acme.storage=/config/secrets/acme.json" # you'll need to create this file ahead of time with chmod 600 perms - "--certificatesresolvers.cloudflare.acme.dnschallenge.delayBeforeCheck=42" - "--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1,1.0.0.1" volumes: - /var/run/docker.sock:/var/run/docker.sock:ro # Needed in R/O mode to allow the docker provider to work - traefik_config:/config # Local store for our LE certs networks: - bridge - traefik_dmz volumes: traefik_config: driver_opts: type: nfs o: addr=your-nfs-ip-or-name,nolock,soft,rw device: :/path/to/your/nfs/export networks: bridge: external: true traefik_dmz: external: true lemmy/docker-compose.yml yaml version: "3.8" services: lemmy-server: image: dessalines/lemmy:0.17.3 configs: - source: lemmy.hjson # A pre created static config file supported by docker swarm target: /config/config.hjson # See https://join-lemmy.org/docs/en/administration/configuration.html deploy: mode: replicated replicas: 1 placement: constraints: - node.role == worker labels: - traefik.enable=true - traefik.docker.network=traefik_dmz # the following roughly translates to the Nginx config from the offical # doc at https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/nginx.conf#L63 - traefik.http.routers.leddit-api.rule=Host(`your-lemmy.domain`) && (PathPrefix(`/api`, `/pictrs`, `/feeds`, `/nodeinfo`, `/.well-known`) || Method(`POST`) || Headers(`accept`, `application/activity+json`) || Headers(`accept`, `application/ld+json; profile="https://www.w3.org/ns/activitystreams"`)) - traefik.http.routers.leddit-api.entrypoints=https - traefik.http.routers.leddit-api.tls=true - traefik.http.routers.leddit-api.tls.certresolver=cloudflare - traefik.http.routers.leddit-api.tls.domains[0].main=your-lemmy.domain - traefik.http.services.leddit-api.loadbalancer.server.port=8536 networks: - traefik_dmz - bridge - app lemmy-ui: image: dessalines/lemmy-ui:0.17.3 environment: # this needs to match the hostname defined in the lemmy service - LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy-server:8536 # set the outside hostname here - LEMMY_UI_LEMMY_EXTERNAL_HOST=your-lemmy.domain - LEMMY_HTTPS=true - NODE_ENV=production deploy: mode: replicated replicas: 1 placement: constraints: - node.role == worker labels: - traefik.enable=true - traefik.docker.network=traefik_dmz - traefik.http.routers.leddit-web.rule=Host(`your-lemmy.domain`) - traefik.http.routers.leddit-web.entrypoints=https - traefik.http.routers.leddit-web.tls=true - traefik.http.routers.leddit-web.tls.certresolver=cloudflare - traefik.http.routers.leddit-web.tls.domains[0].main=your-lemmy.domain - traefik.http.services.leddit-web.loadbalancer.server.port=1234 networks: - traefik_dmz - app pictrs: image: asonix/pictrs environment: - PICTRS__API_KEY=*** user: 991:991 volumes: - pictrs_data:/mnt deploy: mode: replicated replicas: 1 placement: constraints: - node.role == worker networks: - app configs: lemmy.hjson: external: true networks: app: # a stack specific network for limited communication scope (only services on this network can connect to pictrs) internal: true traefik_dmz: external: true bridge: external: true volumes: pictrs_data: driver_opts: type: nfs o: addr=your-nfs-ip-or-name,nolock,soft,rw device: :/path/to/your/nfs/export Since we have multiple manager nodes you can deploy HAProxy and load balance across the manager nodes to their traefik instances using /ping for health checks and a split DNS on your local network to reduce Cloudflare traffic. Using another LE certificate on HAProxy and exposing it externally will allow you to bypass Cloudflare all together if/when is needed. I hope this little(?) guide will be helpful.