Getting timestamp from WAF/Kinesis to Elasticsearch

Question

We're trying to find how to convert the timestamp coming from AWS WAF/Kinesis Firehose into Elasticsearch so that it's type is a date field. When the index mapping gets created it has the timestamp field but it's a type long, even though there seems to be an option for type epoch_millis (which is what the data is).

The Kibana interface says using the mapping api to change the field type but I can't seem to figure this out. The example here shows how to do this with creating new indexes but kinesis is creating/rotating the indexes so we seem to need a way to modify the default.

The field looks like this

  "timestamp": {
    "type": "long"
  },

and the full index definition looks like this but again these get created on a regular basis so we are trying to figure out how to change the default

  "waf-prod-2018-10-05": {
    "mappings": {
      "waf-prod": {
        "properties": {
          "action": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "formatVersion": {
            "type": "long"
          },
          "httpRequest": {
            "properties": {
              "args": { 
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "clientIp": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "country": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "headers": {
                "properties": {
                  "name": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  },
                  "value": {
                    "type": "text",
                    "fields": {
                      "keyword": {
                        "type": "keyword",
                        "ignore_above": 256
                      }
                    }
                  }
                }
              },
              "httpMethod": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "httpVersion": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "uri": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "httpSourceId": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "httpSourceName": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "nonTerminatingMatchingRules": {
            "properties": {
              "action": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "ruleId": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "rateBasedRuleList": {
            "properties": {
              "limitKey": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "maxRateAllowed": {
                "type": "long"
              },
              "rateBasedRuleId": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          },
          "terminatingRuleId": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "terminatingRuleType": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "timestamp": {
            "type": "long"
          },
          "webaclId": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  },

rios0rios0 · Accepted Answer · 2021-02-16 17:52:39Z

The code of the template depends of your ES version. If you are using version 7.x. You need to remove the field (the mapping type field, in your case called by "waf-prod") before the properties and after the mappings. You can try to do this (for example, this is my configuration of ES 7.x):

PUT _template/template_waf-logs
{
  "order": 0,
  "index_patterns": [
    "aws-waf-logs-detected-requests-*"
  ],
  "settings": {
    "index": {
      "number_of_shards": "1",
      "number_of_replicas": "0",
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "properties": {
      "httpRequest": {
        "properties": {
          "clientIp": {
            "type": "keyword",
            "fields": {
              "keyword": {
                "type": "ip"
              }
            }
          }
        }
      },
      "timestamp": {
        "type": "date",
        "format": "epoch_millis"
      }
    }
  }
}

Check de documentation of AWS here: https://aws.amazon.com/blogs/security/how-to-analyze-aws-waf-logs-using-amazon-elasticsearch-service/
Update your knowledge with the answer of the ES community here: https://discuss.elastic.co/t/root-mapping-definition-has-unsupported-parameters-when-creating-custom-index/240690

Explanation to add the mapping type: https://discuss.elastic.co/t/common-error-root-mapping-definition-has-unsupported-parameters/63300/2
Explained by ES version: https://stackoverflow.com/a/39290979/10746857

Unheilig · Accepted Answer · 2018-11-16 02:29:58Z

-2

Just add the timestamp format to the mapping:

"timestamp": {
    "type": "date",
    "format": "epoch_millis"
}

edited Nov 16, 2018 at 2:29

Unheilig

16.3k193 gold badges71 silver badges101 bronze badges

answered Nov 16, 2018 at 2:07

bbambbam sh2ee2

932 silver badges9 bronze badges

Collectives™ on Stack Overflow

Getting timestamp from WAF/Kinesis to Elasticsearch

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related