Parsing JSON array in Fluent-Bit

Question

I am able to parse JSON like string that looks something like this:

{"response":"{\"key1\": \"12345\", \"key2\": \"abc\", \"time\": \"2006-07-28T13:22:04Z\"}"}

My parser:

[PARSER]
    Name        objectParser
    Format      json
    Decode_field_as escaped response

However sometimes I have an array of JSON object that I am unable to parse with built in parsers.

{"response":"[{\"name\": \"john\", \"salary\": \"5\", \"time\": \"2006-07-28T13:22:04Z\"},{\"name\": \"david\", \"salary\": \"7\", \"time\": \"2006-07-28T13:22:04Z\"}]"}

I have tried to use a couple of parsers; however, it seems that all built in parsers are made for one object only instead of an array of objects. I might be wrong, but I was unable to find anything on the internet.

Dean Meehan · Accepted Answer · 2025-05-06 21:54:22Z

It seems that the JSON parser used by Fluent-bit does not support JSON Arrays as the root object for parsing. I did try a couple of different examples but there is a hacky workaround for your use case if you need it.

The following example uses the new YAML format for configuration which I'd recommend. It is setup that:

Creates a new json parser called json (Optional)
inputs.tail :: Reads from test.log the following string (for testing purposes) {"response":"[{\"name\": \"john\", \"salary\": \"5\", \"time\": \"2006-07-28T13:22:04Z\"},{\"name\": \"david\", \"salary\": \"7\", \"time\": \"2006-07-28T13:22:04Z\"}]"}
1. Has the tail parser set to json for the default key log
2. Has a log processor that calls the lua code in code
  1. This code looks for all logs with the response field (string) starting with [{ which indicates that the string is a list and Adds {"my_list": before and } after the string to make the string a valid JSON list, enclosed as an object.
filters.parser :: You then parse again the contents of this response field
outputs.stdout :: Output the log to stdout

parsers:
  - name: json
    format: json
pipeline:
  inputs:
    - name: tail
      path: /test.log
      read_from_head: true
      parser: json
      key: log
      processors:
        logs:
          - name: lua
            call: cb_json_array
            code: |
              function cb_json_array(tag, timestamp, record)
                 local new_record = record
                 if record.response and type(record.response) == "string" then
                     -- Check if response starts with [{"
                     if string.match(record.response, "^%[%{") then
                         -- Add {"my_list": before and } after the string
                         new_record.response = "{\"list\":" .. record.response .. "}"
                     end
                 else
                     new_record = record
                 end
                 return 1, timestamp, new_record
              end
  filters:
    - name: parser
      match: '*'
      key_name: response
      parser: json
  outputs:
    - name: stdout
      match: '*'

Thank you. I haven't tried using it this way, I will try and give you a feedback here. Also, I have realized that lua script can actually help here, so I am using a script to convert any kind of a record received by fluent-bit.

Collectives™ on Stack Overflow

Parsing JSON array in Fluent-Bit

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related