Adding data from Azure Event Hub to Blob

Question

I've got a number of process that throw data an an Azure Event Hub.

I've a Stream Analytics Job that takes each message and pushes it top a blob in a specified folder.

This all seems fin until I look at, or try and query, the data in the blob file (it is being stored, FYI, in an Azure Gen2 Storage Account). The problem is that each message pushed to the Evnet Hub is a complete JSON document.

{
  "Key1": {
    "SubKey1" : 99,
    "SubKey2" : "A String"
  },
  "Key2" : "And so on"
}

However, the process of pushing the items to Blob Storage just results in

{"Key1": {"SubKey1":1, "SubKey2": "Boo"}, "Key2: Thing"}
{"Key1": {"SubKey1":2, "SubKey2": "Blah"}, "Key2: Another Thing"}

Meaning I have multiple top-level items when what I need is something more like

[
  {"Key1": {"SubKey1":1, "SubKey2": "Boo"}, "Key2: Thing"}, 
  {"Key1": {"SubKey1":2, "SubKey2": "Blah"}, "Key2: Another Thing"}
]

I know I'd doing it wring, I just don't know how to stop doing it wrong.

Am I asking to much of the Streaming Analytics Job? Should I be using something else to store my data in the blob store? Or is there a magic spell that I haven't yet found?

Provide the input data you are passing via azure event hub and the query being used in stream analytics job, to get the output and what is the output you are expecting? — Rajesh Mopati
– Rajesh Mopati, Commented May 5, 2023 at 5:45

PramodValavala · Accepted Answer · 2023-05-15 13:26:53Z

0

The output that you are seeing is JSON Lines, which is one JSON object per line. This is the default format and is recommended since objects can be appended to a file quickly.

Parsing such a file is simple since you just must read each line and deserialize it, instead of doing the whole file.

You could however change the format to JSON Array which will give you what you need.

Refer to this official doc for more information.

answered May 15, 2023 at 13:26

PramodValavala

6,6701 gold badge16 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Adding data from Azure Event Hub to Blob

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related