Create multiple pyspark dataframes from csv file

Question

I have a csv file which is in below format.

key_string,query

abc,"select * from abc"

pqr,"select * from pqr"

xyz,"select * from xyz"

These tables are in Hive. I want to create dataframes for eg: abc_df,pqr_df and so on. I can be adding more queries to the csv in future. How can I create multiple dataframes in pyspark using for loop or any other technique? I tried following code but its not working: df is I have read the above csv file

x=""
y=[]
for i in df.rdd.collect():
    x= i[0] + "_df"
    x = spark.sql(i[1])
    y.append(x)
print(y)`

Pls suggest next steps

What do you mean by it’s not working? What is your expected outcome, and what did you obtain from your code? — mck
– mck, Commented Dec 15, 2020 at 6:04
@mck I just want to create dataframes from the queries available in csv files with key_string_df as dataframe name — PPK
– PPK, Commented Dec 15, 2020 at 7:04
it's a bad idea to have variables as variable names. This is what a dictionary is built for. Do you want a dictionary instead? like {'key_string_df': dataframe, ...} — mck
– mck, Commented Dec 15, 2020 at 7:06

mck · Accepted Answer · 2020-12-15 07:08:01Z

1

I'd suggest using a dictionary for this purpose:

y = dict()
for i in df.rdd.collect():
    y[i[0] + "_df"] = spark.sql(i[1])

If you want to get the dataframes, you can use, for example,

y['abc_df'].show()

answered Dec 15, 2020 at 7:08

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

PPK Over a year ago

Thanks @mck. that helped a lot. One more thing how can I directly access/use abc_df.show() instead of y['abc_df'].show()

mck Over a year ago

I don't advice using variables as variable names. See my comment in your question. That's why I suggested using dictionary. Is there a problem with using dictionary?

Collectives™ on Stack Overflow

Create multiple pyspark dataframes from csv file

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related