How to configure NGINX to avoid 502 Bad Gateway when communicates with a Docker container?

Question

Hi, my case is that after having set up a new Droplet with an Ubuntu 18.04 server and have followed the DO tutorials to:

Install Nginx
Secure Nginx with Let’s Encrypt
Install Jenkins
Configure Jenkins with an SSL behind an Nginx reverse proxy
Set up continuous integration pipeline with Jenkins (this doc says for Ubuntu 16.04)
Install docker

So the goal is to manage the deployment of a React application to my testdomain.com later, by now, I just want to create the dist folder generated, after the ‘npm build’, within the /var/lib/jenkins/workspace/ , just that. For the purpose, I will run it in a Jenkins pipeline with docker.

So by now, I’m able to access my jenkins.testdomain.com alright, trigger the pipeline to start the process after pushing to my repo, and start to run the stages; but it’s here when start to fail nginx, when the pipeline reaches the Deliver phase (let’s read ‘npm build’ phase). It’s at this point reading the Jenkins console output I see when it gets stuck in that phase to eventually show a 502 Bad Gateway. After this, I will need to make a systemctl restart Jenkins to have access again. After restarting the pipeline resume the work and seems to get the job done :/

In the /var/log/nginx/error.log for nginx I can read:

*1 connect() failed (111: Connection refused) while connecting to upstream, client: 85.146.85.194, server: jenkins.testdomain.com, request: “GET /job/Basic%20NodeJS-React%20app/8/console HTTP/1.1”, upstream: “https://127.0.0.1:8080/job/Basic NodeJS-React app/8/console”, host: “jenkins.testdomain.com”, referrer: “https://jenkins.testdomain.com/job/Basic NodeJS-React app/8/”

*1 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 85.146.85.194, server: jenkins.testdomain.com, request: “GET /favicon.ico HTTP/1.1”, upstream: “https://127.0.0.1:8080/favicon.ico”, host: “jenkins.testdomain.com”, referrer: “https://jenkins.testdomain.com/job/Basic NodeJS-React app/8/console” …

In the Jenkinsfile of my node-js-react app, the agent looks like this (pass tests from nginx -t)

pipeline {
    agent {
        docker {
            image 'node:6-alpine' 
            args '-p 3000:80' 
        }
    }
    environment {
            CI = 'true'
    }
    stages {
            // Build, Test, and Deliver stages
    }
}

And my jenkins.testdomain.com configuration (/etc/nginx/sites-available/jenkins.testdomain.com) is like this:

server {
    listen 80;

    root /var/www/jenkins.testdomain.com/html;
    server_name jenkins.testdomain.com www.jenkins.testdomain.com;

    location / {
      proxy_set_header    Host $host;
      proxy_set_header    X-Real-IP $remote_addr;
      proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;

      # Fix the "It appears that your reverse proxy set up is broken" error.
      proxy_pass          http://localhost:8080;

      # High timeout for testing
      proxy_connect_timeout 1200s;
      proxy_send_timeout 1200s;
      proxy_read_timeout 1200s;

      proxy_redirect      http://localhost:8080 https://jenkins.testdomain.com;

      # Required for new HTTP-based CLI
      proxy_http_version 1.1;
      proxy_request_buffering off;
  
      # Required for HTTP-based CLI to work over SSL
      proxy_buffering off;
    }

    # Certbot auto-generated lines...
}

Please, any help on this it would really helpful, after 2-3 days I’ve tried several things to avoid that error, but it gets back at some point.

Thanks in advance!

jlbb · Accepted Answer

OK just add an update that some days after my latest post, I realized that the main and only reason the server was going down was a lack of resources in the droplet. So I was using a droplet with 1GB of RAM, 25GB HD, etc… (the most basic one), so I chose to update it to use at least 2GB of RAM and indeed, that made it work as I was expecting. Everything until now works fine and that issue didn’t happen again.

Hope it helps if someone experiences the same issue.

Bobby Iliev · Answer

Hello,

This is an interesting case. Does this happen every time you run the pipeline or intermittently? Also, does it fail straight away or does it take some time and then it locks up?

I could suggest maybe checking your syslog for some extra information:

tail -100 /var/log/syslog

Also, maybe have a look at the Jenkins error log itself at: /var/log/jenkins/jenkins.log for any errors there as well.

Regards, Bobby

jlbb · Answer

Hi,

After checking it several times, it looks like that the react-nodejs project I’m trying, it’s intermittently failing. Some times complete the whole task successfully, but in other cases, it just seems to stop the Deploy process and after some seconds (let’s say around one minute as most) it ends up showing the 502 Bad Gateway error.

Other nodejs-react project I’m trying (with my own custom webpack, which it creates the build locally without issues), it always gets the error in the ‘npm build’ phase or sometimes even earlier (‘npm install’ phase). But again, when I restart Jenkins it resumes and gets the job done!

I’ve already tried playing around with the memory from Jenkins if that would be the case, but with no difference perceived with these changes: JAVA_ARGS="-Xmx256m" or JAVA_ARGS="-Xmx2048m -XX:MaxPermSize=512m"

File /var/log/jenkins/jenkins.log is empty.
/var/log/syslog shows some possible error, I cannot really figure out what’s going with it:

kernel: [ 1716.509477] docker0: port 1(vethcf5108c) entered blocking state kernel: [ 1716.509479] docker0: port 1(vethcf5108c) entered disabled state kernel: [ 1716.509606] device vethcf5108c entered promiscuous mode systemd-timesyncd[535]: Network configuration changed, trying to establish connection. networkd-dispatcher[779]: WARNING:Unknown index 13 seen, reloading interface list kernel: [ 1716.522790] IPv6: ADDRCONF(NETDEV_UP): vethcf5108c: link is not ready systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). systemd-udevd[5472]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. systemd-udevd[5472]: Could not generate persistent MAC address for vethc118367: No such file or directory systemd-udevd[5474]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. systemd-udevd[5474]: Could not generate persistent MAC address for vethcf5108c: No such file or directory systemd-timesyncd[535]: Network configuration changed, trying to establish connection. systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). containerd[811]: time="2019-10-14T10:48:04.960106015+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/b1ae788b18bd4dfd5fd7870cc194dd0bbee8591ee5402846cc4030945d8794de/shim.sock" debug=false pid=5485 systemd-timesyncd[535]: Network configuration changed, trying to establish connection. systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). kernel: [ 1717.167230] eth0: renamed from vethc118367 kernel: [ 1717.168525] IPv6: ADDRCONF(NETDEV_CHANGE): vethcf5108c: link becomes ready kernel: [ 1717.168564] docker0: port 1(vethcf5108c) entered blocking state kernel: [ 1717.168566] docker0: port 1(vethcf5108c) entered forwarding state systemd-networkd[592]: vethcf5108c: Gained carrier systemd-networkd[592]: docker0: Gained carrier systemd-timesyncd[535]: Network configuration changed, trying to establish connection. systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). systemd-networkd[592]: vethcf5108c: Gained IPv6LL systemd-timesyncd[535]: Network configuration changed, trying to establish connection. systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). kernel: [ 1719.454959] [UFW BLOCK] IN=eth0 OUT= MAC=MAC_ADDRESS SRC=89.248.162.168 DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=252 ID=42701 PROTO=TCP SPT=40602 DPT=3580 WINDOW=1024 RES=0x00 SYN URGP=0

Thanks in advance.

Question

How to configure NGINX to avoid 502 Bad Gateway when communicates with a Docker container?

Become a contributor for community

DigitalOcean Documentation

Resources for startups and SMBs

Get our newsletter

The developer cloud

Get started for free