How to add custom index using ingest node pipeline?

Question

Is it possible to create conditional indexing by using ingest node pipelines? I feel this could be done by the script processor but can someone tell if this is possible?

I am in a scenario where I should decide which is a better way to do custom indexing. I can mention conditions in the metricbeat.yml /filebeat.yml files to get this done. But is this the best way to do custom indexing? There is no logstash in my elastic stack

output.elasticsearch:
          indices:
            - index: "metricbeat-dev-%{[agent.version]}-%{+yyyy.MM.dd}"
              when.equals:
                kubernetes.namespace: "dev"

This is how I have implemented custom indexing in metric/filebeat right now. I have like 20+ namespaces in my Kubernetes cluster. Please help in suggesting if this could be done by ingest node pipeline or not

Frederik Struck-Schøning · Accepted Answer · 2023-09-13 13:07:23Z

3

Yes, You can achived this by ingest pipeline Set Processor. Ingest Pipeline support accessing of metadata fields and you can access / update index name using _index field name.

Below is sample Ingest Pipeline which will update index name when namespace is dev:

[
  {
    "set": {
      "field": "_index",
      "value": "metricbeat-dev",
      "if": "ctx.kubernetes?.namespace== 'dev'"
    }
  }
]

Upadte 1: append agent version to index name. I ahve consider agent version feild name as agent.version

[
      {
        "set": {
          "field": "_index",
          "value": "metricbeat-dev-{{agent.version}}",
          "if": "ctx.kubernetes?.namespace== 'dev'"
        }
      }
    ]

edited Sep 13, 2023 at 13:07

Frederik Struck-Schøning

13.3k9 gold badges64 silver badges70 bronze badges

answered Dec 8, 2021 at 8:31

Sagar Patel

5,5161 gold badge11 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

sidharth vijayakumar Over a year ago

When i implement this in my metricbeat.yml I am getting the index as .ds-metricbeat-dev any idea why it is creating a data stream?

Sagar Patel Over a year ago

@sidharthvijayakumar I am not exectly sure but it might be happening as your index name is starting with metricbeat- which is matching with the existing ILM and index template. Please raise seperate question for this.

Sagar Patel Over a year ago

@sidharthvijayakumar Please accept my answer if it is helped you.

sidharth vijayakumar Over a year ago

This is not exactly what i want but still works but its not creating an alias and stuff so am not able to implement ilm

Sagar Patel Over a year ago

@sidharthvijayakumar are you trying with metricbeat- index name or other custom inex name

|

Collectives™ on Stack Overflow

How to add custom index using ingest node pipeline?

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related