How to convert RDD[(String, String)] into RDD[Array[String]]?

Question

I am trying to append filename to each record in the file. I thought if the RDD is Array it would have been easy for me to do it.

Some help with converting RDD type or solving this problem would be much appreciated!

In (String, String) type

scala> myRDD.first()(1)    
scala><console>:24: error: (String, String) does not take parametersmyRDD.first()(1)

In Array(string)

scala> myRDD.first()(1)    
scala> res1: String = abcdefgh

My function:

def appendKeyToValue(x: Array[Array[String]){
    for (i<-0 to (x.length - 1)) {
        var key = x(i)(0)
        val pattern = new Regex("\\.")
        val key2 = pattern replaceAllIn(key1,"|")
        var tempvalue = x(i)(1)
        val finalval = tempvalue.split("\n")
        for (ab <-0 to (finalval.length -1)){
            val result = (I am trying to append filename to each record in the filekey2+"|"+finalval(ab))
            }  
        }
}

Jacek Laskowski · Accepted Answer · 2015-09-15 15:45:39Z

8

If you have a RDD[(String, String)], you can access the first tuple field of the first tuple by calling

val firstTupleField: String = myRDD.first()._1

If you want to convert a RDD[(String, String)] into a RDD[Array[String]] you can do the following

val arrayRDD: RDD[Array[String]] = myRDD.map(x => Array(x._1, x._2))

You may also employ a partial function to destructure the tuples:

val arrayRDD: RDD[Array[String]] = myRDD.map { case (a,b) => Array(a, b) }

edited Sep 15, 2015 at 15:45

Jacek Laskowski

75k28 gold badges253 silver badges440 bronze badges

answered Sep 15, 2015 at 10:15

Till Rohrmann

13.4k2 gold badges27 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

WoodChopper Over a year ago

Both answers provides solution to my query. Cheers!

Till Rohrmann Over a year ago

Glad that I could help :-)

Collectives™ on Stack Overflow

How to convert RDD[(String, String)] into RDD[Array[String]]?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related