Using Scala script in Spark

Question

We are planning to using Scala on Spark to make computations. Just want to know what is the best way to execute Scala in Spark ; Scala as Script (or) Scala as Application. Is there any advantage/disadvantage between these 2 methods?

As mentioned here it is possible to execute Scala as Script. I am trying to skip the compilation process using sbt so that I can use Scala as script just like we will use Python

user1314742 · Accepted Answer · 2016-05-04 08:18:07Z

2

I suppose you mean by scala as script the scala REPL comes with spark (spark-shell) and scala application is the standlaone appliction packaged by sbt or maven.

Use scala shell (spark-shell) to test your algoriothm/implementation . So it should be used as staging phase.
When you have tested your implementation you should put it in a standalone application, package it and deliever a "fat jar" that could be submited by using spark-submit

Hope this is clear enough

answered May 4, 2016 at 8:18

user1314742

2,9243 gold badges29 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Raj Over a year ago

I have updated my question. I just want to skip the manual compilation process using sbt ; so that I can execute scala as script similar to Python

user1314742 Over a year ago

That is the same thing mentioned in my answer. Most of the time you use some dependencies in your application so you need to package them within your application to launch it on a cluster. for Scala scripts ,IMHO, could not be delivered in production environment

Collectives™ on Stack Overflow

Using Scala script in Spark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related