how to create dynamic dataframe name in pyspark

Question

how to create dynamic dataframe name in pyspark here I am not able to create new dataframe using below code it will give me only last dataframe name, I need All dataframe name

for prime2 in pdf2:
    ol2 =  Bucket_path + prime2['S3_File_with_Path']
    t = 1
    sd = {}  
    testR = "df" + str(t)
    print("testR",testR)
    sd[testR] = spark.read.format("parquet").load(ol2).cache() 
    t = t + 1

Rayan Ral · Accepted Answer · 2020-06-22 17:25:24Z

2

Seems like you're creating dict inside the loop, so getting a dict with only one (last) entry. Try changing code to something like this:

sd = {}  
for prime2 in pdf2:
    ol2 =  Bucket_path + prime2['S3_File_with_Path']
    t = 1
    testR = "df" + str(t)
    print("testR",testR)
    df = spark.read.format("parquet").load(ol2).cache() 
    sd[testR] = df
    t = t + 1 

# sd dict is available here, all the dataframes are inside
print(len(sd))

edited Jun 22, 2020 at 17:25

answered Jun 22, 2020 at 15:36

Rayan Ral

1,8692 gold badges17 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Rayan Ral Over a year ago

hey @Ajinkya, was that answer helpful?

Collectives™ on Stack Overflow

how to create dynamic dataframe name in pyspark

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related