Cannot parse CSV file with Logstash

Question

I have an issue with importing a CSV file into ElasticSearch using logstash for further processing in Kibana.

This is my logstash config file:

input {
    file {
        path => ["/absolute_path_to_file/export.csv"]
        start_position => beginning 
        ignore_older => 0 
        sincedb_path => "/dev/null"
    }
}
#filter {
#   csv {
#       columns => [
#           "id",
#           "cislo_smlouvy",
#           "zdroj",
#           "produkt",
#           "sjednani",
#           "datum_odeslani",
#           "cas_odeslani",
#           "pojistovna",
#           "tarif",
#           "pojistnik",
#           "telefon",
#           "predmet_pojisteni",
#           "rz",
#           "pocatek_pojisteni",
#           "rocni_pojistne",
#           "urgence",
#           "stav"
#       ]
#       separator => ";"
#       remove_field => ["message"]
#   }
#}
output {
#   elasticsearch {
#       hosts => "localhost:9200"
#       index => "smlouvy"
#   }
    stdout {
        codec => rubydebug
    }
}

And an excerpt from my CSV file:

"id";"číslo smlouvy";"zdroj";"produkt";"sjednání";"datum odeslaní";"čas odeslání";"pojišťovna";"tarif";"pojistník";"pojistnik telefon";"předmět pojištění";"rz";"počátek";"roční pojistné";"urgence";"stav"
"114951";"6132681255";"SRO";"POV";;"1.6.2016";"12:28";"csob";"csob-2";"BB TEST";"721666333";"Škoda Favorit";"NENÍ";"2.6.2016 00:00";"4657,00";;"TEST"
"114950";;"POV";"POV";"VO Bukvicova";"1.6.2016";"12:16";"csob";"csob-2";"BB BB";"721000111";"BMW X3";"NENÍ";"3.6.2016 00:00";"5550,00";;"TEST"

I am calling this command:
sudo logstash -f /absolute_path_to_file/logstash.conf --vebrose

With the following output:

starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 2
Registering file input {:path=>["/absolute_path_to_file/export.csv"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
Pipeline main started

After a while of doing nothing, I shut it down:

^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Closing inputs {:level=>:info}
Closed inputs {:level=>:info}
Input plugins stopped! Will shutdown filter/output workers. {:level=>:info}
Pipeline main has been shutdown

Possibly relevant version info:

logstash 2.3.2
logstash-input-file (2.2.5)
logstash-filter-csv (2.1.3)
logstash-output-elasticsearch (2.6.2)
logstash-output-stdout (2.0.6)
logstash-codec-rubydebug (2.0.7)

I have read all the documentation I could find and tried to replicate a lot of logstash.conf examples from GitHub, but with no luck. Any help with what I am missing?

If I uncomment your csv filter, I do get the proper output with the expected data from your CSV file. — Val
– Val, Commented Jun 2, 2016 at 6:33
And I don't. I had a feeling that the config is right, it seems to be some other problem with logstash itself... — Patrick Kusebauch
– Patrick Kusebauch, Commented Jun 2, 2016 at 8:58
You might try to use the exact same headers with the proper accents, etc and see if that changes something. — Val
– Val, Commented Jun 2, 2016 at 8:59
I tried that, but it didn't help. I started to seriously doubt my logstash installation, even the tutorial apachelog setup didn't work for me. I am currently trying to use a dockerized ELK stack to see if it's any help.. — Patrick Kusebauch
– Patrick Kusebauch, Commented Jun 2, 2016 at 10:19
Just tried in dockerized environment github.com/deviantony/docker-elk and it still didn't work — Patrick Kusebauch
– Patrick Kusebauch, Commented Jun 2, 2016 at 10:41

Patrick Kusebauch · Accepted Answer · 2016-06-02 11:03:13Z

1

So I finally found out the problem. It was with the input CSV file.

The CSV file had \r for new line and logstash default is \n.

BTW: You cannot set \r as the delimiter in logstash file input filter config, so I had to convert the CSV file into one with \n

answered Jun 2, 2016 at 11:03

Patrick Kusebauch

873 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Val Over a year ago

Oh, good catch. I didn't notice since I copy/pasted from your post.

Patrick Kusebauch Over a year ago

Yeah, neither did I. I hate I had to spent hours debugging such issue.

Val Over a year ago

Just be more skeptical, whenever using a file produced by an MS product ;-)

Collectives™ on Stack Overflow

Cannot parse CSV file with Logstash

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related