how to create broadcast variable in spark 2(java)?

Question

In Spark 1 we can use the below code to create a Spark broadcast variable:

SparkConf conf = new SparkConf();
conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer");

JavaSparkContext sc = new JavaSparkContext("local", "JavaAPISuite", conf);

ArrayList<Strin'valuesg> sampleList = new ArrayList<String>();
sampleList.add("value");

final Broadcast<ArrayList<String> broadcastVar = sc.broadcast(sampleList);

How can we do the same in Spark 2 using the API shown below?

SparkSession sc = SparkSession.setappName("SparkApp").getorcreate();
sc.sparkcontext().broadcast(T value, scala.reflect.ClassTag<T> evidence$11)

hi cricket_007...the link shows the Spark 1 type...not the SparkSession of Spark2 — Kiran Balakrishnan
– Kiran Balakrishnan, Commented Aug 27, 2017 at 13:18
What do you mean? You need a SparkContext to broadcast. You get the context from the Spark session — OneCricketeer
– OneCricketeer, Commented Aug 27, 2017 at 14:20
You copied the function wrong, by the way. There's one parameter, the value... The implicit argument can be ignored broadcast[T](value: T)(implicit arg0: ClassTag[T]): Broadcast[T] — OneCricketeer
– OneCricketeer, Commented Aug 27, 2017 at 14:24
@cricket_007...as per spark api documentation for Java ,i needs to input parameters for creating a broadcast variable..as given below <T> Broadcast<T> broadcast(T value, scala.reflect.ClassTag<T> evidence$11) Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions. — Kiran Balakrishnan
– Kiran Balakrishnan, Commented Aug 28, 2017 at 5:02

kofucii · Accepted Answer · 2017-10-31 10:33:23Z

4

For example if you want to broadcast class named Test it should implement java.io.Serializable and then you do:

import scala.reflect.ClassTag;

ClassTag<Test> classTagTest = scala.reflect.ClassTag$.MODULE$.apply(Test.class);
Broadcast<Test> broadcastTest = sc.sparkcontext().broadcast(new Test(), classTagTest);

answered Oct 31, 2017 at 10:33

kofucii

7,71313 gold badges55 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

how to create broadcast variable in spark 2(java)?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related