I have a simple PySpark code on Databricks which reads the data from a bucket, few minor transformations and writes to delta table.
Currently following below steps. I have hardcoded catalog/schema/table names for now. I need to dynamically replace catalog/schema/table based on parameters. With PySpark, we can easily replace values, is there anyway we can replace values for <catalog_name>.<schema_name>.<table_name> used in final SQL insert, based on input parameters?
- Read input data and create dataframe
- basic transformation
- create temp table from dataframe using createOrReplaceTempView()
- write to delta table as below
df = spark.read.option("header", "true") .schema(schema_name).csv(source_path)
df.createOrReplaceTempView(temp_table)
# how can we replace the names based on input parameter/variable
insert into <"name_of_catalog">.<"name_of_schema">.<"name_of_table">
select * from temp_table;
CREATE WIDGET TEXT database_name DEFAULT "default"; SHOW TABLES IN ${database_name}- you can try something similar I guess...