ElasticSearch on docker - 2nd instance kills the first instance

Question

I'm trying to run multiple versions of ElasticSearch at the same time, should be easy. Here are my commands:

docker run -d --rm -p 9250:9200 -p 9350:9300 --name es_5_3_3_integration -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:5.3.3
docker run -d --rm -p 9251:9200 -p 9351:9300 --name es_5_4_3_integration -e "xpack.security.enabled=false" docker.elastic.co/elasticsearch/elasticsearch:5.4.3

The first docker starts up great. The 2nd docker starts, but at the cost of killing the first docker. If I run it without the -d I don't get any info back to the UI about why the docker stopped.

How much RAM did you give to docker? ES 5 starts with 2GB of heap by default (see jvm.options) so if your Docker doesn't have at least 4GB the first instance gets killed indeed. — Val
– Val, Commented Nov 28, 2017 at 20:42
That's probably it. How can I tell the docker to use only, say, 500M of memory? — jhilden
– jhilden, Commented Nov 28, 2017 at 20:44
You should probably start ES with less heap than 2GB per server, I'd say. — Val
– Val, Commented Nov 28, 2017 at 20:44
you were right @Val, adding -e ES_JAVA_OPTS="-Xms200m -Xmx200m" fixed it. Want to write up an answer and I'll give you the points? :) — jhilden
– jhilden, Commented Nov 28, 2017 at 20:48

jhilden · Accepted Answer · 2017-11-28 20:57:15Z

By default, ES on docker tries to take 2G of memory. So 2 dockers was trying to take up 4G of memory, which my machine didn't have.

The solution: limit the amount of memory each ES instance tried to take to 200mb using the following switch -e ES_JAVA_OPTS="-Xms200m -Xmx200m"

Full, working commands for 4 concurrent dockers:

docker run -d --rm -p 9250:9200 -p 9350:9300 --name es_5_3_3_integration -e "xpack.security.enabled=false" -e ES_JAVA_OPTS="-Xms200m -Xmx200m" docker.elastic.co/elasticsearch/elasticsearch:5.3.3
docker run -d --rm -p 9251:9200 -p 9351:9300 --name es_5_4_3_integration -e "xpack.security.enabled=false" -e ES_JAVA_OPTS="-Xms200m -Xmx200m" docker.elastic.co/elasticsearch/elasticsearch:5.4.3
docker run -d --rm -p 9252:9200 -p 9352:9300 --name es_5_5_3_integration -e "xpack.security.enabled=false" -e ES_JAVA_OPTS="-Xms200m -Xmx200m" docker.elastic.co/elasticsearch/elasticsearch:5.5.3
docker run -d --rm -p 9253:9200 -p 9353:9300 --name es_5_6_4_integration -e "xpack.security.enabled=false" -e ES_JAVA_OPTS="-Xms200m -Xmx200m" docker.elastic.co/elasticsearch/elasticsearch:5.6.4

Thank you to @Val who really answered this question in the comments.

Guillaume Husta · Accepted Answer · 2017-11-28 22:06:12Z

If this is a lack of memory problem, you can check if your container was OOMKilled (OOM).

First check if the exit code of the container is 137 = (128+9) Container received a SIGKILL.
You can test it with docker ps -a or

docker inspect --format='{{.State.ExitCode}}' $INSTANCE_ID

Then you can check the state of the container with :

docker inspect --format='{{.State.OOMKilled}}' $INSTANCE_ID

If it returns true, it was a OOM problem.

Further details at https://docs.docker.com/engine/reference/run/#user-memory-constraints .
Extract :

By default, kernel kills processes in a container if an out-of-memory (OOM) error occurs. To change this behaviour, use the --oom-kill-disable option. Only disable the OOM killer on containers where you have also set the -m/--memory option. If the -m flag is not set, this can result in the host running out of memory and require killing the host’s system processes to free memory.

Collectives™ on Stack Overflow

ElasticSearch on docker - 2nd instance kills the first instance

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related