Logstash: MySQL to Elasticsearch (large table)?

Question

I am attempting to import a rather chunky database into Elasticsearch. It has 4m rows across 2 columns (VARCHAR(250) & INT(20)).

When I run the logstash.conf file to import the database into Elasticsearch and I add a LIMIT 0,100 in my SQL command the command runs without any problems. All of the rows end up returned by Logstash in Terminal and then I can see them in the relevant node in Elasticsearch.

When I try to run all of the rows through Logstash at once, it outputs:

Settings: Default pipeline workers: 1 Pipeline Main Started

And nothing more happens.

How do I add such a large table into Elasticsearch?

Here's my logstash.conf script:

input{

  jdbc {
jdbc_driver_library => "/opt/logstash/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar"

jdbc_driver_class => "com.mysql.jdbc.Driver"

jdbc_connection_string => "jdbc:mysql://<ip number>:3306/database"

jdbc_validate_connection => true

jdbc_user => "elastic"

jdbc_password => "password"

schedule => "* * * * *"

    statement => "name, id from master_table"

    }

}

 output

 {
  elasticsearch
    {

    index => "search"
    document_type => "name"
    document_id => "%{id}"
    hosts => "127.0.0.1:9200"
}stdout { codec => json_lines }
}

dadoonet · Accepted Answer · 2016-06-27 00:40:24Z

3

I would set the fetch_size to something like 10000 documents. Here I think that it tries to load all records in memory which can take hours and probably won't fit.

See https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_dealing_with_large_result_sets

answered Jun 27, 2016 at 0:40

dadoonet

14.6k3 gold badges46 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Alex Meraz · Accepted Answer · 2017-06-08 23:16:08Z

2

input {
    jdbc {
        ...
        jdbc_page_size => 100000 
        jdbc_paging_enabled => true
    }
}
...

Res:

[logstash.inputs.jdbc] SELECT count(*) AS count FROM (SELECT * FROM my_table) AS t1 LIMIT 1

[logstash.inputs.jdbc] SELECT * FROM (SELECT * FROM my_table) AS t1 LIMIT 100000 OFFSET 0

[logstash.inputs.jdbc]SELECT * FROM (SELECT * FROM my_table) AS t1 LIMIT 100000 OFFSET 100000

...

answered Jun 8, 2017 at 23:16

Alex Meraz

213 bronze badges

Collectives™ on Stack Overflow

Logstash: MySQL to Elasticsearch (large table)?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related