Using conditional statements in map() in Spark with Java

Question

I'd like to use a conditional statement in a map function. For example,

JavaPairRDD<Integer, Long> rdd1 = sc.parallelize(Arrays.asList(1, 2, 1, 0)).zipWithIndex();
rdd1.map(x -> if x._1 == 2 return x._1*x._1 else return x._1).foreach(x -> System.out.println(x));

should print [(1,0), (4, 1), (1, 2), (0, 3)]. So, what's the proper way of using the map function in such way ?

MigSena · Accepted Answer · 2017-04-21 21:31:43Z

1

You can write your code like this :

JavaPairRDD<Integer, Long> rdd1 = sc.parallelize(Arrays.asList(1, 2, 1,   0)).zipWithIndex();
rdd1.mapToPair(x -> 
        {
            if (x._1 == 2) 
                return new Tuple2<Integer, Long>(x._1*x._1, x._2); 
            else 
                return new Tuple2<Integer, Long>(x._1, x._2); 
        }).foreach(x -> System.out.println(x));

answered Apr 21, 2017 at 21:31

MigSena

2182 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

SpiderRico Over a year ago

Thanks! Was really helpful.

leoconco · Accepted Answer · 2017-04-21 20:32:59Z

0

JavaPairRDD does not define a map() function, but a mapValue() function which can only change the value on a pair and not the key, you could try using another RDD, or, if using that one, don't put values you would want to change in the Key position, but in an Object in the value.

answered Apr 21, 2017 at 20:32

leoconco

2833 silver badges15 bronze badges

Collectives™ on Stack Overflow

Using conditional statements in map() in Spark with Java

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related