spark query execution time

Question

I have a local hadoop single node and hive installed and I have some hive tables stored in hdfs. Then I configure Hive with MySQL Metastore. And now I installed spark and Im doing some queries over hive tables like this (in scala):

var hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
result = hiveContext.sql("SELECT * FROM USERS");
result.show

Do you know how to configure spark to show to the execution time of the query? Because for default it is not showing..

Possible duplicate of this stackoverflow.com/questions/34629313/… — charles gomes
– charles gomes, Commented Apr 3, 2016 at 18:36
OMG, isn't there anyone who finally doesn't suggest to visit a web browser to get the execution time?! can't believe it. This has to also be possible programmatically. — Nadjib Mami
– Nadjib Mami, Commented Oct 19, 2016 at 8:03

Tyrone321 · Accepted Answer · 2018-05-11 10:03:09Z

2

Use spark.time().

var hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) result = hiveContext.sql("SELECT * FROM USERS"); spark.time(result.show)

https://db-blog.web.cern.ch/blog/luca-canali/2017-03-measuring-apache-spark-workload-metrics-performance-troubleshooting

edited May 11, 2018 at 10:03

answered May 11, 2018 at 9:34

Tyrone321

1,95218 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

spark query execution time

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related