pyspark save dataframe to hive table using variable in the name

Question

I am using databrick pyspark for coding wondering how could I pass the variable value to the name of the table which I want to save in the Azure

I am able to use this if that is the fix table name

result.write.format("parquet").mode("overwrite").saveAsTable("result0911")

but I hope to do this

Having a variable: time = datetime.today() the value of that is "2019-09-11 12:10:48.969602"

I want to first format the "time" to "20190911121048" which is good for the name of a table (I need the time also, because people may save the records more than once per day)

then use that value "20190911121048" table name which I want to save the table name should be "result_20190911121048"

any suggestion? Thank you the following does not work

result.write.format("parquet").mode("overwrite").saveAsTable("result_%s time")

I am able to use {t=time.strftime("%Y%m%d%H%M")} to reformat the time but still not abke to put the value of t in the name. — CloverCeline
– CloverCeline, Commented Sep 11, 2019 at 12:49

ravi malhotra · Accepted Answer · 2019-09-11 13:45:54Z

2

Say you have 2 variables having values

time = "20190911121048"
table_name = 'result'

Then you can make table name as :

final_table_name = '{}_{}'.format(table_name, time)

result.write.format("parquet").mode("overwrite").saveAsTable(final_table_name)

answered Sep 11, 2019 at 13:45

ravi malhotra

7335 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ravi malhotra Over a year ago

@CloverCeline please let me know if this is what you were looking for and please accept the answer

CloverCeline · Accepted Answer · 2019-09-12 16:49:41Z

1

Thank you for the input. I am able to use this:

time = datetime.today()
t=time.strftime("%Y%m%d%H%M")
result_name ="recommendation_%s" % (t)
final_recon.write.format("parquet").mode("overwrite").saveAsTable(result_name)

answered Sep 12, 2019 at 16:49

CloverCeline

5334 silver badges22 bronze badges

Collectives™ on Stack Overflow

pyspark save dataframe to hive table using variable in the name

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related