1

I am importing a syslog using logstash (version 1.5.3) into elasticsearch(version 1.7.1) using the following configuration.

input{
  file {
    path => "somepath\*.log"
  }
}
filter{
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:message_hostname} %{DATA:message_program}(?:\[%{POSINT:message_pid}\])?: %{GREEDYDATA:user_message}" }
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
}
output {
  elasticsearch{
    cluster => somecluster 
    host => localhost 
    index => "logindex-%{+YYYY-MM-dd}"
  }
}

My index is created based on the current date and time i.e. logindex-2015-08-07.

I want to create the index based on the date syslog_timestamp and not the current date using the above format {+YYYY-MM-dd}

So If the log had a timestamp 2015-01-01, my index should be created as logindex-2015-01-01 and not logindex-2015-08-07

EDIT

Log Input Used:

Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message

Logstash Debug Output

←[36mfilter received {:event=>{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]:(root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, :level=>:debug, :file=>"(eval)", :line=>"69", :method=>"filter_func"}
 ←[0m
 ←[36mRunning grok filter {:event=>#<LogStash::Event:0x166d6250 @metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root)
CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, "host"]}>>, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-1.0.0/lib/logstash/filters/grok.rb", :line=>"283", :method=>"filter"}
←[0m
←[36mRegexp match object {:names=>["SYSLOGTIMESTAMP:message_timestamp", "SYSLOGHOST:message_hostname", "DATA:message_program", "POSINT:message_pid", "GREEDYDATA:user_message"], :captures=>["Jul 27 07:49:01", "Server1", "CRON", "21009", "(root) CMD LTest Message\r"], :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=>"179", :method=>"match_and_capture"}
←[0m
←[36mfilters/LogStash::Filters::Grok: adding value to field {:field=>"received_at", :value=>["%{@timestamp}"], :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/util/decorators.rb", :line=>"28", :method=>"add_fields"}

←[0m
←[36mEvent now:  {:event=>#<LogStash::Event:0x166d6250 @metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "host"], "message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009","user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message"], "message_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_timestamp"], "message_hostname"=>[{"message"=>"Jul27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_hostname"], "message_program"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTestMessage\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_program"], "message_pid"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_pid"], "user_message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "user_message"], "@timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "@timestamp"], "received_at"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01","message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "received_at"]}>>, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-1.0.0/lib/logstash/filters/grok.rb", :line=>"303", :method=>"filter"}

←[0m
←[36mDate filter: received event {:type=>nil, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter
-date-1.0.0/lib/logstash/filters/date.rb", :line=>"206", :method=>"filter"}
←[0m←[36mDate filter looking for field {:type=>nil, :field=>"syslog_timestamp", :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/
jruby/1.9/gems/logstash-filter-date-1.0.0/lib/logstash/filters/date.rb", :line=>
"209", :method=>"filter"}
←[0m
←[36moutput received {:event=>{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]:(root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, :level=>:debug, :file=>"(eval)", :line=>"76", :method=>"output_func"}
←[0m
←[36mFlushing output {:outgoing_count=>1, :time_since_last_flush=>22.048, :outgoing_events=>{nil=>[["index", {:_id=>nil, :_index=>"%index-2015-08-14", :_type=>"logs", :_routing=>nil},
#<LogStash::Event:0x166d6250 
@metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={"retry_count"=>0}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @metadata={"retry_count"=>0}, 
@accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>sage\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r","received_at"=>"2015-08-14T07:34:53.215Z"}, "host"], "message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message"], "message_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_timestamp"], "message_hostname"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_hostname"], "message_program"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_program"], "message_pid"=>[{"message"=>"Jul 27 07:49:01 Server1CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_pid"], "user_message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "user_message"], "@timestamp"=>[{"message"=>"Jul 2707:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "@timestamp"], "received_at"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "received_at"], "type"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r","received_at"=>"2015-08-14T07:34:53.215Z"}, "type"], "syslog_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "syslog_timestamp"]}>>]]}, :batch_timeout=>1, :force=>nil, :final=>nil, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/stud-0.0.20/lib/stud/buffer.rb", :line=>"207", :method=>"buffer_flush"}

←[0m{              "message" => "Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r",
             "@version" => "1",
           "@timestamp" => "2015-08-14T07:34:53.215Z",
                 "host" => "HOST-LT",
    "message_timestamp" => "Jul 27 07:49:01",
     "message_hostname" => "Server1",
      "message_program" => "CRON",
          "message_pid" => "21009",
         "user_message" => "(root) CMD LTest Message\r",
          "received_at" => "2015-08-14T07:34:53.215Z"
}
Improve this question

1 Answer 1

Reset to default
2

The problem might be that the locale on your machine is different from the locale used to produce the log. So you should specify the locale in your date filter, like this:

...
filter{
    grok {
        ...
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      locale => "en"
    }
}
...

UPDATE

Based on your log output above (very helpful!!), your date filter should work on the message_timestamp field and not syslog_timestamp (which doesn't exist)

Improve this answer
Sign up to request clarification or add additional context in comments.

11 Comments

The date field is coming out correct. The issue is that I want my ES index (index => "logindex-%{+YYYY-MM-dd}") to be created based on the syslog_timestamp with format YYYY-MM-dd and not the current system date
Weird, as that's what the date filter does by default, i.e. it copies the matched date to the @timestamp field. How does your @timestamp field look like?
In ES, @timestamp comes out as the current date.Like "@timestamp": "2015-08-14T06:28:02.018Z" for a log inserted today even though the log is of July 27th. Not sure if this adds any benefit but I have this add_field => [ "received_at", "%{@timestamp}" ] in grok which I omitted as it just adds another field from @timestamp.
Oh, and what version of Logstash are you using?
version 1.5.3 for Logstash
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.