Mapping error in Elasticsearch?

Question

I have an index named againagain-* which was created by logstash. I did the

curl command

before I ran the config file into logstash. However after all these steps, I went into Discover tab in Kibana but all the strings are still deemed as True under the "Analzyed Fields". Is there something wrong with my mapping?

P.S I did it before and after I ran my config file too.

    curl –XPUT http://localhost:5601/againagain -d ‘
{
 "mappings" : {
  "_default_" : {
   "properties" : {
"service" : { "type" : "integer" },
"rule" : { "type" : "integer" },
"ICMP Type" : { "type" : "integer" },
"ICMP Code" : { "type" : "integer" },
"ip_offset" : { "type" : "integer" },
"ip_id" : { "type" : "integer" },
"ip_len" : { "type" : "integer" },
"Confidence Level" : { "type" : "integer" },
"fragments_dropped" : { "type" : "integer" },
"Severity" : { "type" : "integer" },
"serial_num" : { "type" : "integer" },
"during_sec" : { "type" : "integer" },
"Attack info" : {"type": "string", "index" : "not_analyzed" },
"peer gateway" : {"type": "string", "index" : "not_analyzed" },
"SmartDefense Profile" : {"type": "string", "index" : "not_analyzed" },
"FollowUp" : {"type": "string", "index" : "not_analyzed" },
"attack" : {"type": "string", "index" : "not_analyzed" },
"type" : {"type": "string", "index" : "not_analyzed" },
"Performance Impact" : {"type": "string", "index" : "not_analyzed" },
"reject_category" : {"type": "string", "index" : "not_analyzed" },
"action" : {"type": "string", "index" : "not_analyzed" },
"ICMP" : {"type": "string", "index" : "not_analyzed" },
"inzone" : {"type": "string", "index" : "not_analyzed" },
"dn" : {"type": "string", "index" : "not_analyzed" },
"proto" : {"type": "string", "index" : "not_analyzed" },
"dst" : {"type": "string", "index" : "not_analyzed" },
"message_info" : {"type": "string", "index" : "not_analyzed" },
"ICMP" : {"type": "string", "index" : "not_analyzed" },
"Severity" : {"type": "string", "index" : "not_analyzed" },
"rule_uid" : {"type": "string", "index" : "not_analyzed" },
"CookieI" : {"type": "string", "index" : "not_analyzed" },
"interface" : {"type": "string", "index" : "not_analyzed" },
"IKE" : {"type": "string", "index" : "not_analyzed" },
"TCP packet out of state" : {"type": "string", "index" : "not_analyzed" },
"service_id" : {"type": "string", "index" : "not_analyzed" },
"vpn_feature_name" : {"type": "string", "index" : "not_analyzed" },
"Protection Type" : {"type": "string", "index" : "not_analyzed" },
"src" : {"type": "string", "index" : "not_analyzed" },
"ip_len" : {"type": "string", "index" : "not_analyzed" },
"fw_subproduct" : {"type": "string", "index" : "not_analyzed" },
"protection_id" : {"type": "string", "index" : "not_analyzed" },
"Protection Name" : {"type": "string", "index" : "not_analyzed" },
"tcp_flags" : {"type": "string", "index" : "not_analyzed" },
"Internal_CA" : {"type": "string", "index" : "not_analyzed" },
"outzone" : {"type": "string", "index" : "not_analyzed" },
"scheme" : {"type": "string", "index" : "not_analyzed" },
"Reason" : {"type": "string", "index" : "not_analyzed" },
"message" : {"type": "string", "index" : "not_analyzed" },
"product" : {"type": "string", "index" : "not_analyzed" },
"Industry Reference" : {"type": "string", "index" : "not_analyzed" }
   }
  }
 }
}
';

The output they gave me in Terminal after this command is <.!DOCTYPE html.><.html.><.head.><.title.><.link rel="stylesheet" href="/styles/main.css".><./head.><.body.><.h1>Not Found<./h1><./body.><./html.">kibana"

You are curling kibana not elasticsearch! Try to curl on your elasticsearch port. The default one is 9200 — eliasah
– eliasah, Commented Aug 20, 2015 at 6:50
Oh that was stupid of me! Do I curl 1st before running my config file? Or... — imbadatcoding
– imbadatcoding, Commented Aug 20, 2015 at 7:21
I have a logstash config file and a sample data. However after running the config file, my sample data fields are considered as analyzed fields which means that strings such as Hello everyone is being broken into Hello and everyone. That's the reason why I'm mapping my variables into all not_analyzed fields. So the question is should I run my config file to upload the data or use the curl command to define the mapping 1st — imbadatcoding
– imbadatcoding, Commented Aug 20, 2015 at 7:25

eliasah · Accepted Answer · 2015-08-20 07:32:48Z

Like i said in the comment, you are trying put your mapping on the kibana port (5601) instead of the elasticsearch port (9200, by the default).

curl -XPUT "http://localhost:9200/againagain/" -d'
{
  "mappings": {
    "_default_": {
      "properties": {
        "service": {
          "type": "integer"
        },
        "rule": {
          "type": "integer"
        },
        "ICMP Type": {
          "type": "integer"
        },
        "ICMP Code": {
          "type": "integer"
        },
        "ip_offset": {
          "type": "integer"
        },
        "ip_id": {
          "type": "integer"
        },
        "ip_len": {
          "type": "string",
          "index": "not_analyzed"
        },
        "Confidence Level": {
          "type": "integer"
        },
        "fragments_dropped": {
          "type": "integer"
        },
        "Severity": {
          "type": "string",
          "index": "not_analyzed"
        },
        "serial_num": {
          "type": "integer"
        },
        "during_sec": {
          "type": "integer"
        },
        "Attack info": {
          "type": "string",
          "index": "not_analyzed"
        },
        "peer gateway": {
          "type": "string",
          "index": "not_analyzed"
        },
        "SmartDefense Profile": {
          "type": "string",
          "index": "not_analyzed"
        },
        "FollowUp": {
          "type": "string",
          "index": "not_analyzed"
        },
        "attack": {
          "type": "string",
          "index": "not_analyzed"
        },
        "type": {
          "type": "string",
          "index": "not_analyzed"
        },
        "Performance Impact": {
          "type": "string",
          "index": "not_analyzed"
        },
        "reject_category": {
          "type": "string",
          "index": "not_analyzed"
        },
        "action": {
          "type": "string",
          "index": "not_analyzed"
        },
        "ICMP": {
          "type": "string",
          "index": "not_analyzed"
        },
        "inzone": {
          "type": "string",
          "index": "not_analyzed"
        },
        "dn": {
          "type": "string",
          "index": "not_analyzed"
        },
        "proto": {
          "type": "string",
          "index": "not_analyzed"
        },
        "dst": {
          "type": "string",
          "index": "not_analyzed"
        },
        "message_info": {
          "type": "string",
          "index": "not_analyzed"
        },
        "rule_uid": {
          "type": "string",
          "index": "not_analyzed"
        },
        "CookieI": {
          "type": "string",
          "index": "not_analyzed"
        },
        "interface": {
          "type": "string",
          "index": "not_analyzed"
        },
        "IKE": {
          "type": "string",
          "index": "not_analyzed"
        },
        "TCP packet out of state": {
          "type": "string",
          "index": "not_analyzed"
        },
        "service_id": {
          "type": "string",
          "index": "not_analyzed"
        },
        "vpn_feature_name": {
          "type": "string",
          "index": "not_analyzed"
        },
        "Protection Type": {
          "type": "string",
          "index": "not_analyzed"
        },
        "src": {
          "type": "string",
          "index": "not_analyzed"
        },
        "fw_subproduct": {
          "type": "string",
          "index": "not_analyzed"
        },
        "protection_id": {
          "type": "string",
          "index": "not_analyzed"
        },
        "Protection Name": {
          "type": "string",
          "index": "not_analyzed"
        },
        "tcp_flags": {
          "type": "string",
          "index": "not_analyzed"
        },
        "Internal_CA": {
          "type": "string",
          "index": "not_analyzed"
        },
        "outzone": {
          "type": "string",
          "index": "not_analyzed"
        },
        "scheme": {
          "type": "string",
          "index": "not_analyzed"
        },
        "Reason": {
          "type": "string",
          "index": "not_analyzed"
        },
        "message": {
          "type": "string",
          "index": "not_analyzed"
        },
        "product": {
          "type": "string",
          "index": "not_analyzed"
        },
        "Industry Reference": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    }
  }
}'

yes I did that but i have two same variables. One of them is analyzed and not_analyzed. The not_analyzed is empty so I can't use Kibana to output them to charts etc

bravosierra99 · Accepted Answer · 2015-08-21 17:15:56Z

0

So your question earlier is spot on. You need your index to be empty first. So curl your mapping and THEN send your data in using logstash. I'm not sure if it's required but you might want to clear your mappings in elasticsearch when you delete your index. I've run into the same issue before. There might be easier methods but I just delete everything, put in my mapping, then import data and I've always been good to go.

answered Aug 21, 2015 at 17:15

bravosierra99

1,3711 gold badge11 silver badges24 bronze badges

Collectives™ on Stack Overflow

Mapping error in Elasticsearch?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related