In my spark application I read ONCE a directory with many CSVs. But, in the DAG I see multiple CSV reads.
- Why the spark reads multiple times the CSVs or it's not a real representation; and actually Spark reads them once.
Spark UI Screenshot:
Add a comment
|
1 Answer
Spark will read them multiple times if the DataFrame is not cached.
val df1 = spark.read.csv("path")
val df2_result = df1.filter(.......).save(......)
val df3_result = df1.map(....).groupBy(...).save(......)
Here df2_result and df3_result both will cause df1 to be rebuilt from csv files. To avoid this you can cache like this. DF1 will built once from csv and the 2nd time it will not be build from files.
val df1 = spark.read.csv("path")
df1.cache()
val df2_result = df1.filter(.......).save(......)
val df3_result = df1.map(....).groupBy(...).save(......)
2 Comments
ggeop
I understand this behavior if I had multiple actions in my DAG. I have only one action. I will add a cache after the read and I will see again the DAG.
ggeop
I find the problem.. I have only one action but I create intermediate dataframes and I assign them in python variables, so the spark creates difference branch in the DAG for these dataframes and reads again the CSVs. So, yes the caching in my case solves the problem and reads only once.