Heads up! Long post and lots of head bashing against the wall.

Context:

I have a written an a python app (Django). I have dockerized the deployment and the compose file has three containers, app, nginx and postgres. I’m currently trying to deploy a demo of it in a VPS running Debian 11. Information below has been redacted (IPs, Domain name, etc.)

Problem:

I keep running into 502 errors. Locally things work very well even with nginx (but running on 80). As I try to deploy this I’m trying to configure nginx the best I can and redirecting http traffic to https and ssl certs. The nginx logs simply say “connect() failed (111: Connection refused) while connecting to upstream, client: 1.2.3.4, server: demo.example.com, request: “GET / HTTP/1.1”, upstream: “http://192.168.0.2:8020/”, host: “demo.example.com””. I have tried just about everything.

What I’ve tried:

  • Adding my server block configs to /etc/nginx/conf.d/default.conf
  • Adding my server block configs to a new file in /etc/nginx/conf.d/app.conf and leaving default at out of box config.
  • Tried putting the above config (default.conf and app.conf) in sites-available (/etc/nginx/sites-available/* not at the same time tho).
  • Recreated /etc/nginx/nginx.conf by copy/pasting out of box nginx.conf and then adding server blocks directly in nginx.conf
  • Running nginx -t inside of the nginx container (Syntax and config were “successful”)
  • Running nginx -T when recreated /etc/nginx/nginx.conf
    • nginx -T when the server blocks where in /etc/nginx/conf.d/* lead me to think that since there were two server listen 80 blocks that I should ensure only one listen 80 block was being read by the container hence the recreated /etc/nginx/nginx.conf from above
  • Restarted container each time a change was made.
  • Changed the user block from nginx (no dice when using nginx as user) to www-data, root and nobody
  • Deleted my entire docker data and redeployed everything a few times.
  • Double checked the upstream block 1,000 times
  • Confirmed upstream block container is running and on the right exposed port Checked access.log and error.log but they were both empty (not sure why, tried cat and tail)
  • Probably forgetting more stuff (6 hours deep in the same error loop by now)

How can you help:

Please take a look at the nginx.conf config below and see if you guys can spot a problem, PLEASE! This is my current /etc/nginx/nginx.conf

`

user www-data;

worker_processes auto;

error_log /var/log/nginx/error.log notice; pid /var/run/nginx.pid;

events { worker_connections 1024; }

http { include /etc/nginx/mime.types; default_type application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

keepalive_timeout  65;

#gzip  on;

upstream djangoapp {
    server app:8020;
}

server {
    listen 80;
    listen [::]:80;
    server_name demo.example.com;

    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name demo.example.com;

    ssl_certificate /etc/letsencrypt/live/demo.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/demo.example.com/privkey.pem;
    #ssl_protocols TLSv1.2 TLSv1.3;
    #ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://djangoapp;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        #proxy_set_header Upgrade $http_upgrade;
        #proxy_set_header Connection keep-alive;
        proxy_redirect off;
    }

    location /static/ {
        autoindex on;
        alias /static/;
    }
}

}

`

  • EDIT: I have also confirmed that both containers are connected to the same docker network (docker network inspect frontend)

  • EDIT 2: Solved my problem. See my comments to @chaospatterns. TLDR there was an uncaught exception in the app but it didn’t cause a crash with the container. Had to dig deep into logs to find it.

  • @lal309OP
    link
    English
    11 year ago

    Yea I always try to dedicate networks to each app and if it’s a full stack app then one for front end (nginx and app) and another for backend (app and database).

    I didn’t think about spinning up the alpine container to troubleshoot so that’s another great pointer for future soul crushing and head bashing sessions!