Convert Spark Java to Spark scala

Question

I am trying to convert my Java code to scala in Spark, but found it very complicated. Is it possible to convert the following Java code to scala? Thanks!

JavaPairRDD<String,Tuple2<String,String>> newDataPair = newRecords.mapToPair(new PairFunction<String, String, Tuple2<String, String>>() {

            private static final long serialVersionUID = 1L;

            @Override
            public Tuple2<String, Tuple2<String, String>> call(String t) throws Exception {
                MyPerson p = (new Gson()).fromJson(t, MyPerson.class);

                String nameAgeKey = p.getName() + "_" + p.getAge() ;

                Tuple2<String, String> value = new Tuple2<String, String>(p.getNationality(), t);
                Tuple2<String, Tuple2<String, String>> kvp =
                    new Tuple2<String, Tuple2<String, String>>(nameAgeKey.toLowerCase(), value);
                return kvp;
            }
        });

I tried the following, but I am sure I have missed many things. And actually it is not clear to me how to do the override function in scala ... Please suggest or share some examples. Thank you!

val newDataPair = newRecords.mapToPair(new PairFunction<String, String, Tuple2<String, String>>() {

        @Override
        public val call(String t) throws Exception {
            val p = (new Gson()).fromJson(t, MyPerson.class);
            val nameAgeKey = p.getName() + "_" + p.getAge() ;
            val value = new Tuple2<String, String>(p.getNationality(), t);
            val kvp =
                new Tuple2<String, Tuple2<String, String>>(nameAgeKey.toLowerCase(), value);
            return kvp;
        }
    });

Please show us what you have tried

yjshen
– yjshen

2015-05-12 00:48:42 +00:00
Commented May 12, 2015 at 0:48 — yjshen
– yjshen, Commented May 12, 2015 at 0:48
This is what I tried:

Edamame
– Edamame

2015-05-12 01:41:55 +00:00
Commented May 12, 2015 at 1:41 — Edamame
– Edamame, Commented May 12, 2015 at 1:41

maasg · Accepted Answer · 2015-05-12 02:57:20Z

3

Literal translations from Spark-Java to Spark-Scala typically don't work because Spark-Java introduces many artifacts to cope with the limited type system in Java. Examples in this case: mapToPair in Java is just map in Scala. Tuple2 has a more terse syntax (a,b)

Applying that (and some more) to the snippet:

val newDataPair = newRecords.map{t =>                
    val p = (new Gson()).fromJson(t, classOf[MyPerson])
    val nameAgeKey = p.getName + "_" + p.getAge
    val value = (p.getNationality(), t)
    (nameAgeKey.toLowerCase(), value)
}

It could be made a bit more concise but I wanted to keep the same structure as the Java counterpart to facilitate the understanding of it.

edited May 12, 2015 at 2:57

answered May 12, 2015 at 2:47

maasg

37.5k14 gold badges91 silver badges117 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Edamame Over a year ago

I got " identifier expected but 'class' found" at MyPerson.class on the line: val p = (new Gson()).fromJson(t, MyPerson.class) ... what am I missing? Thanks!

maasg Over a year ago

@Edamame oops - fixed - another java vs scala thing :-) --- btw, I'm not familiar on how Guava's Gson would interop with Scala - might work as-is. Otherwise, you'll have to research further.

Collectives™ on Stack Overflow

Convert Spark Java to Spark scala

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related