Spark tasks execution series

Question

I am running a Spark application on the driver.

Is is simple as follow

val count=0; 
val test_dataframe =//extrenal frame
count=test.count();
println("The count of frame is " + count);

My question is that if the third line is always executed after computing the count of frame. Is it possible the driver first run the println command, before executing the dataframe and its count?

how will you print the count without performing count operation on dataframe? — vatsal mevada
– vatsal mevada, Commented Sep 11, 2017 at 13:49
I saw that. However that didn't make any sense to me to print just zero. What if you put that println statement before count=test.count() this line? — vatsal mevada
– vatsal mevada, Commented Sep 12, 2017 at 3:40
my question is that possible println run without processing datafraem and then printing 0? — Luckylukee
– Luckylukee, Commented Sep 12, 2017 at 5:18
I asssume that instead of test.count you meant test_dataframe.count. As your declared dataframe name is test_dataframe and not test. — vatsal mevada
– vatsal mevada, Commented Sep 12, 2017 at 20:08

vatsal mevada · Accepted Answer · 2017-09-13 07:57:37Z

1

No it is not possible that driver will execute println before test.count() in above mentioned code as count is a tearminal operation and call to a terminal operation forces spark to perform computation before moving on.

If you want async count then here is a code snippet which works:

var future = test.rdd.countAsync
println("The count before future evaluation: " + count)
count = future.get
println("The count after future evaluation: " + count)

Note that countAsync action is not available directly on dataframe. It can be performed on RDD.

edited Sep 13, 2017 at 7:57

answered Sep 12, 2017 at 20:05

vatsal mevada

5,7368 gold badges48 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Luckylukee Over a year ago

I feel not, "test.count" would be an Asynchronous operation in the nature!

vatsal mevada Over a year ago

count is a synchronous action by default. There are several async actions supported by spark. If you want asyc behaviour then you need to explicitly mention that in your code.check this class: spark.apache.org/docs/2.2.0/api/java/index.html?org/apache/…

vatsal mevada Over a year ago

@Luckylukee does above explanation clears your doubt? Also I have updated the answer with a code snippet for async count.

Collectives™ on Stack Overflow

Spark tasks execution series

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related