Grafana doesn't work with nginx proxy_pass variable

Question

We're running grafana and nginx in docker swarm, and proxying the url /foobar/ to the swarm instance of grafana. Using this guide, this works with the following config:

# nginx config
server {
    resolver 127.0.0.11 valid=30s;

    ...

    location /foobar/ {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://grafana:3000/;
        proxy_next_upstream error timeout http_502;
    }
}

# docker-compose
  grafana:
    image: ${REGISTRY}foo/grafana:${IMAGE_VERSION}
    networks:
      - foo
    volumes:
      - grafana:/var/lib/grafana
    environment:
      - GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/foobar/

However, this causes nginx to die on startup if the grafana service is not available. So to resolve this, we use a variable for the proxy_pass directive and change it to this:

server {
    resolver 127.0.0.11 valid=30s;

    ...

    location /foobar/ {
        set $grafana http://grafana:3000;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass $grafana/;
        # proxy_pass http://grafana:3000/;
        proxy_next_upstream error timeout http_502;
    }

}

However, this causes grafana to reject the request somehow. I can verify that grafana is actually receiving the request (using GF_SERVER_ROUTER_LOGGING=true), and it claims the status is 200 ok, however the only thing I see on the page is

If you're seeing this Grafana has failed to load its application files 

1. This could be caused by your reverse proxy settings.

2. If you host grafana under subpath make sure your grafana.ini root_path setting includes subpath

3. If you have a local dev build make sure you build frontend using: npm run dev, npm run watch, or npm run build

4. Sometimes restarting grafana-server can help

Why does grafana behave like this, and how can I set up the proxy pass such that nginx can start up without trying to resolve the grafana URL if it happens to be down?

Tarun Lalwani · Accepted Answer · 2019-06-25 04:01:47Z

3

When using variables complete URL is your responsibility in a proxy pass

    location /foobar/ {
        set $grafana http://grafana:3000;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass $grafana$request_uri;
        # proxy_pass http://grafana:3000/;
        proxy_next_upstream error timeout http_502;
    }

In case the base path is different then you will need to use regular expression to send part of the path

    location ~ /foobar/(.*) {
        set $grafana http://grafana:3000;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass $grafana/$1;
        proxy_next_upstream error timeout http_502;
    }

answered Jun 25, 2019 at 4:01

Tarun Lalwani

147k11 gold badges218 silver badges279 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Paul Richter Over a year ago

I do not understand why, but the second version is the one that worked.

Collectives™ on Stack Overflow

Grafana doesn't work with nginx proxy_pass variable

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related