I’ve seen a couple of instances around asking if it’s possible to self host Lemmy fronted by Traefik, figured I can share my test setup with the community. This configuration is still in testing phase but seems to work in a federated configuration.

The Design

To run in an elastic and highly available way (where possible with my current hardware) I’ve opted to use Docker Swarm as it’s one of the easiest ways to get started with multi-node deployment. There are a few limitations with Swarm and I’m planning on moving to a K3s cluster in the future.

For added protection I’ve chosen to front my services with Cloudflare for the time being, there’s a definite tradeoff of privacy vs security here and since I’m not able to afford traffic scrubbing center and deal with any ISP issues for now it’ll be used.

The Setup

A few prerequisites are needed for this type of deployment although I think it should be fairly easy to adapt it to a single node setup.

  • Docker Swarm - 2x Manger nodes 2x Worker nodes
  • NFS - TrueNAS, used to store LE certs and pictrs blobs
  • Cloudflare’d domain - optional but makes life much easier
  • PostgreSQL Database - A dedicated postgresql VM for persistent database

The Config

I’ll skip some of the basic setup configurations since there are plenty of guides out there and focus on the docker compose files for the cloudflare, traefik, and lemmy stacks.

Starting with the simplest, you can follow any guide to create a new tunnel to use for this setup, the compose file should look something like this:

cloudflare/docker-compose.yml

version: "3.8"
services:
  cloudflare:
    image: cloudflare/cloudflared:2023.5.1-amd64  # always hardcode your versions to avoid unexpected changes
    deploy:
      mode: global  # in swarm we will deploy one instance per manager node (more nodes more redudnacy/LB)
      placement:
        constraints:
          - node.role == manager
    networks:
      - bridge  # a default created network allowing connectivity to the rest of the network (and internet to connect upstream)
      - traefik_dmz  # an overlay internal private network for LB'd services
    command:
      - tunnel
      - --no-autoupdate
      - run
      - --token=***  # replace with the token issued by cloudflare
networks:
  bridge:
    external: true
  traefik_dmz:
    external: true

important: Once you’ve created your tunnal add a public host pointing your-lemmy.domain to https://traefik and under TLS enable the No TLS Verify option. Since both cloudflare and traefik services are on a common network cloudflare can access the internal service VIP by calling traefik (matching the service name) traefik/docker-compose.yml

version: '3.8'
services:
  traefik:
    image: traefik:v2.10
    environment:
     - "CF_DNS_API_TOKEN=***"  # your cloudflare API token with zone read and DNS edit permissions
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host
      - target: 8082
        published: 8082
        protocol: tcp
        mode: host
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik_dmz
        - traefik.http.routers.traefik-dashboard.rule=HostRegexp(`{subhost:your-hostname-pattern[\d]+}.homelab`)  # To access the dashboard you can use your hosts' FQDN pattern and access https://your-hostname-pattern1.homelab 
        - traefik.http.routers.traefik-dashboard.entrypoints=https
        - traefik.http.routers.traefik-dashboard.tls=true
        - traefik.http.routers.traefik-dashboard.service=api@internal
        - traefik.http.services.traefik.loadbalancer.server.port=8080
    command:
      - "--log=true"  # System logs good for debugging
      - "--log.level=WARN"
      - "--accesslog=false"  # Access logs good for debugging but so noisy
      - "--accesslog.format=json"
      - "--accesslog.fields.defaultmode=keep"
      - "--accesslog.fields.headers.defaultmode=keep"
      - "--accesslog.fields.headers.names.Authorization=drop"
      - "--api=true"  # enable for dashboard access
      - "--api.dashboard=true"
      - "--ping=true"  # enable for external LB health checks
      - "--ping.entrypoint=ping"
      - "--serverstransport.insecureskipverify=true"
      - "--global.checknewversion=false"
      - "--global.sendanonymoususage=false"
      - "--entrypoints.http=true"
      - "--entrypoints.http.address=:80"
      - "--entrypoints.http.http.redirections.entrypoint.to=https"
      - "--entrypoints.http.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.https=true"
      - "--entrypoints.https.address=:443"
      - "--entrypoints.https.forwardedheaders.insecure=true"
      - "--entrypoints.https.forwardedheaders.trustedips=10.0.0.0/8"  # limit the x-forwarded-for header trust to your external LB 
      - "--entrypoints.ping=true"
      - "--entrypoints.ping.address=:8082"
      - "--providers.docker=true"  # This will allow us to auto detect configured services and forward traffic
      - "--providers.docker.exposedbydefault=false"
      - "--providers.docker.swarmmode=true"
      - "--certificatesresolvers.cloudflare.acme.dnschallenge=true"  # LetsEncrypt using cloudflare DNS challenge
      - "--certificatesresolvers.cloudflare.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.cloudflare.acme.caserver=https://acme-v02.api.letsencrypt.org/directory"
      - "[email protected]"
      - "--certificatesresolvers.cloudflare.acme.storage=/config/secrets/acme.json"  # you'll need to create this file ahead of time with chmod 600 perms
      - "--certificatesresolvers.cloudflare.acme.dnschallenge.delayBeforeCheck=42"
      - "--certificatesresolvers.cloudflare.acme.dnschallenge.resolvers=1.1.1.1,1.0.0.1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro  # Needed in R/O mode to allow the docker provider to work
      - traefik_config:/config  # Local store for our LE certs
    networks:
      - bridge
      - traefik_dmz
volumes:
  traefik_config:
    driver_opts:
      type: nfs
      o: addr=your-nfs-ip-or-name,nolock,soft,rw
      device: :/path/to/your/nfs/export
networks:
  bridge:
    external: true
  traefik_dmz:
    external: true

lemmy/docker-compose.yml

version: "3.8"
services:
  lemmy-server:
    image: dessalines/lemmy:0.17.3
    configs:
      - source: lemmy.hjson  # A pre created static config file supported by docker swarm
        target: /config/config.hjson  # See https://join-lemmy.org/docs/en/administration/configuration.html
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.role == worker
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik_dmz
          # the following roughly translates to the Nginx config from the offical
          # doc at https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/nginx.conf#L63
        - traefik.http.routers.leddit-api.rule=Host(`your-lemmy.domain`) && (PathPrefix(`/api`, `/pictrs`, `/feeds`, `/nodeinfo`, `/.well-known`) ||  Method(`POST`) ||  Headers(`accept`, `application/activity+json`) ||  Headers(`accept`, `application/ld+json; profile="https://www.w3.org/ns/activitystreams"`))
        - traefik.http.routers.leddit-api.entrypoints=https
        - traefik.http.routers.leddit-api.tls=true
        - traefik.http.routers.leddit-api.tls.certresolver=cloudflare
        - traefik.http.routers.leddit-api.tls.domains[0].main=your-lemmy.domain
        - traefik.http.services.leddit-api.loadbalancer.server.port=8536
    networks:
      - traefik_dmz
      - bridge
      - app
  lemmy-ui:
    image: dessalines/lemmy-ui:0.17.3
    environment:
      # this needs to match the hostname defined in the lemmy service
      - LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy-server:8536
      # set the outside hostname here
      - LEMMY_UI_LEMMY_EXTERNAL_HOST=your-lemmy.domain
      - LEMMY_HTTPS=true
      - NODE_ENV=production
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.role == worker
      labels:
        - traefik.enable=true
        - traefik.docker.network=traefik_dmz
        - traefik.http.routers.leddit-web.rule=Host(`your-lemmy.domain`)
        - traefik.http.routers.leddit-web.entrypoints=https
        - traefik.http.routers.leddit-web.tls=true
        - traefik.http.routers.leddit-web.tls.certresolver=cloudflare
        - traefik.http.routers.leddit-web.tls.domains[0].main=your-lemmy.domain
        - traefik.http.services.leddit-web.loadbalancer.server.port=1234
    networks:
      - traefik_dmz
      - app
  pictrs:
    image: asonix/pictrs
    environment:
      - PICTRS__API_KEY=***
    user: 991:991
    volumes:
      - pictrs_data:/mnt
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.role == worker
    networks:
      - app
configs:
  lemmy.hjson:
    external: true
networks:
  app:  # a stack specific network for limited communication scope (only services on this network can connect to pictrs)
    internal: true
  traefik_dmz:
    external: true
  bridge:
    external: true
volumes:
  pictrs_data:
    driver_opts:
      type: nfs
      o: addr=your-nfs-ip-or-name,nolock,soft,rw
      device: :/path/to/your/nfs/export

Since we have multiple manager nodes you can deploy HAProxy and load balance across the manager nodes to their traefik instances using /ping for health checks and a split DNS on your local network to reduce Cloudflare traffic. Using another LE certificate on HAProxy and exposing it externally will allow you to bypass Cloudflare all together if/when is needed.

I hope this little(?) guide will be helpful.

  • @3v4lu8English
    1•2 years ago

    deleted by creator