Read Json Kafka message without schema

Question

Currently we are working on a real time data feeds having Json data.

While reading the examples from - https://sparkbyexamples.com/spark/spark-streaming-with-kafka/

It looks like we need a schema for kafka json message.

Is there any other way to process data without schema ?

What kind of processing do you want to perform without a schema? — Emiliano Martinez
– Emiliano Martinez, Commented Feb 25, 2022 at 12:26
The data is in the Value field as a string. There you have the data from the topic — Emiliano Martinez
– Emiliano Martinez, Commented Feb 25, 2022 at 12:38
The Value field is binary from the topic. we need to deserialize it and it requires schema to deserialize. — Khilesh Chauhan
– Khilesh Chauhan, Commented Feb 25, 2022 at 12:41
Not necessarily you just need to convert value part in string and then writestrem into destination location — Mahesh Gupta
– Mahesh Gupta, Commented Feb 25, 2022 at 14:56

Mahesh Gupta · Accepted Answer · 2022-02-25 15:02:22Z

2

try below code after running the zookeeper, Kafka server and other required service.

df = spark \
        .readStream \
        .format("kafka") \
        .option("kafka.bootstrap.servers", kafka_bootstrap_servers) \
        .option("subscribe", kafka_topic_name) \
        .option("startingOffsets", "latest")\
        .load()  #earliest

print("Printing Schema of transaction_detail_df: ")
df.printSchema()

transaction_detail_df1 = df.selectExpr("CAST(value AS STRING)")

trans_detail_write_stream = transaction_detail_df1 \
    .writeStream \
    .trigger(processingTime='2 seconds') \
    .option("truncate", "false") \
    .format("console") \
    .start()

trans_detail_write_stream.awaitTermination()

just change the basic configuration, you would be able to see the output

answered Feb 25, 2022 at 15:02

Mahesh Gupta

1,90214 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Khilesh Chauhan Over a year ago

Thanks, This helps to save the json in single column called value. How to extract data from fields inside json ?

Mahesh Gupta Over a year ago

You can explode the dataframe columns based on requirement

Khilesh Chauhan Over a year ago

Thanks, This worked with work around. Since saving the json value this way makes the schema as single string column where 'explode' cannot be applied. However writing as intermediate file as text and reading as json worked.

OneCricketeer · Accepted Answer · 2022-02-26 00:34:55Z

0

You can use get_json_object SparkSQL function to parse data out of JSON string data without defining any additional schema.

You can simply use cast function to deserialize the binary key/value, as the example shows

answered Feb 26, 2022 at 0:34

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Collectives™ on Stack Overflow

Read Json Kafka message without schema

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related