How to stream data from SQL Table with Apache Spark with Databricks

Question

I am attempting to stream from sql table using the following:

my_sales =  spark.read.jdbc(jdbcUrl, dbo.table)

static = spark.read.format("csv").load(my_sales)
dataSchema = static.schema

I am trying to read in the data from the table with the following:

rawdf = (spark.readStream 
      .format("csv") \
      .option("maxFilesPerTrigger", 1) \
      .schema(dataSchema) \
      .csv(dataPath)
           )

I am using the following to write the data to the following location

saveloc = '/mnt/raw/streaminglocation/'


streamingQuery = (
  rawdf
  .writeStream
  .format("csv")
  .outputMode("append")
  .option("checkpointLocation", f"{saveloc}/_checkpoints")
  .option("mergeSchema", "true")
  .start(saveloc)
)

However this failing.

Is it possible to stream from a SQL table?

Ged · Accepted Answer · 2022-02-26 10:48:07Z

1

This is not possible. JDBC sources are not supported for Spark Structured Streaming.

Not convinced of the upfront coding either.

Use CDC with Kafka, or materialized updateable views with CDC with KAFKA, or Debezium.

edited Feb 26, 2022 at 10:48

answered Feb 26, 2022 at 10:31

Ged

18.5k8 gold badges53 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to stream data from SQL Table with Apache Spark with Databricks

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related