Define elasticsearch mapping during Docker build

Question

I have a MySQL, Logstash, and ES setup but I need to set some fields to keyword type instead of text. I've read that it is not possible to do this in Logstash (logstash.conf) and so it needs to be done in ES. I've followed a similar question here and slightly modified it to PUT a mapping but I have got this error: "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [es.path.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings",

I am using docker-compose to start all the services at once under the same network, and so the mapping must be specified before logstash ports the data to ES. (Mapping can't be changed on a non-empty index).

I have seen other questions and they do seem a bit old so I wanted to ask if there is a better approach to doing this now.

My mapping.json

{
    "mappings": {
        "properties": {
            "authors": {"type": "keyword"},
            "tags": {"type": "keyword"}
        }
    }
}

Dockerfile

FROM elasticsearch:7.5.1

COPY ./docker-entrypoint.sh .

COPY ./mapping.json .

RUN mkdir /data && chown -R elasticsearch:elasticsearch /data && echo 'es.path.data: /data' >> config/elasticsearch.yml && echo 'path.data: /data' >> config/elasticsearch.yml

ADD https://raw.githubusercontent.com/vishnubob/wait-for-it/e1f115e4ca285c3c24e847c4dd4be955e0ed51c2/wait-for-it.sh /utils/wait-for-it.sh

# Copy the files you may need and your insert script

RUN ./docker-entrypoint.sh elasticsearch -p /tmp/epid & /bin/bash /utils/wait-for-it.sh -t 0 localhost:9200 -- curl -X PUT 'http://localhost:9200/cnas_publications' -d @./mapping.json; kill $(cat /tmp/epid) && wait $(cat /tmp/epid); exit 0;

Edit: I've used the docker-entrypoint.sh from the official repo here

you should make an API call the mappings endpoint with your mapping.json — Lupanoide
– Lupanoide, Commented Jan 15, 2020 at 12:11
@Lupanoide I added some context above. I know that I need to call the API, but I need to call it before Logstash ports the data to ES. I'm using the official ES Docker image and it seems that just a simple cURL'ing in the Dockerfile isn't working — Steve Hemmingsen
– Steve Hemmingsen, Commented Jan 15, 2020 at 12:45
if you need to put mapping with logstash you you use template instruction to logstash elastic.co/blog/logstash_lesson_elasticsearch_mapping — Lupanoide
– Lupanoide, Commented Jan 15, 2020 at 13:27
please read here elastic.co/guide/en/logstash/current/… template, manage_template, template_name — Lupanoide
– Lupanoide, Commented Jan 15, 2020 at 13:34
@Lupanoide Thanks for point me in the right direction. Strangely others were saying that I should do it make a manual API call to ES rather than using Logstash — Steve Hemmingsen
– Steve Hemmingsen, Commented Jan 16, 2020 at 2:56

Steve Hemmingsen · Accepted Answer · 2020-01-16 03:00:57Z

0

It seems that I was mistaken, and it is actually possible to define the mapping in Logstash. Assuming you're using the official elasticsearch image, create a ES template and make a volume with it to the logstash container.

Here's a sample of my output of my logstash.conf

output {
  stdout { codec => "rubydebug" }
  elasticsearch {
    hosts => "http://elasticsearch:9200"
    index => "test"
    template => "/logstash/mapping.json"
    template_name => "mapping"
    document_id => "%{[@metadata][_id]}"
  }
}

and don't forget to set index_patterns in your ES template.

answered Jan 16, 2020 at 3:00

Steve Hemmingsen

4038 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

dgiffone Over a year ago

Please note that at the moment of writing logstash elasticsearch output plugin supports only the legacy index templates. New index templates are not supported causing a Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL... error. Further details here and here

Collectives™ on Stack Overflow

Define elasticsearch mapping during Docker build

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related