Elasticsearch cluster not working in swarm (stack) , but works with compose

Question

i have a elasticsearch & kibana docker compose file, which works using docker compose up command, but doesn't work with docker stack deploy command. I have tried using network with overlay driver as well. When i check for cluster health using curl localhost:9200/_cluster/health, i get following error:

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

version: '3.4'

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    hostname: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - data01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    hostname: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    hostname: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.15.2
    hostname: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: overlay
    internal: true

Ravi Soni · Accepted Answer · 2021-11-20 13:23:06Z

1

Removing the port did the trick.

Not sure how the port was causing the issue. If anyone knows details, please share.

Here's the updated working compose:

version: '3.4'

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    hostname: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - data01:/usr/share/elasticsearch/data
    networks:
      - elastic

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    hostname: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - data02:/usr/share/elasticsearch/data
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.2
    hostname: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - data03:/usr/share/elasticsearch/data
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:7.15.2
    hostname: kib01
    ports:
      - 5601:5601
    environment:
      ELASTICSEARCH_URL: http://es01:9200
      ELASTICSEARCH_HOSTS: '["http://es01:9200","http://es02:9200","http://es03:9200"]'
    networks:
      - elastic

volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local

networks:
  elastic:
    driver: overlay
    internal: true

answered Nov 20, 2021 at 13:23

Ravi Soni

1,3631 gold badge11 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Chris Becke Over a year ago

There seems to be a lot of cargo culting when it comes to compose files on swarm (and on compose) where unnecessary port, container_name and other directives are added. Generally these things are a bad idea. For service to service comms ports don't need to be published as the services can see each other on their private overlay network. Publishing ports that are not required to accept traffic from outside the swarm causes a potential conflict as only one service on the entire swarm can use the port.

Mark Rajcok Over a year ago

Exposing port 9200 adds the swarm ingress network to each es service, so now there are two 10.x.x.x IP addresses assigned to each es service (and virtual IPs as well). ES discovery seems to try and use both networks, which fails, as documented rather well here. The solution was to use network.public_host. You might want port 9200 exposed so other programs can connect to ES using its API (instead of Kibana) and run a query.

Collectives™ on Stack Overflow

Elasticsearch cluster not working in swarm (stack) , but works with compose

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related