value lookup is not a member of org.apache.spark.rdd.RDD[(String, String)]

Question

I have got a problem when I tired to compile my scala program with SBT. I have import the class I need .Here is part of my code.

import java.io.File
import java.io.FileWriter
import java.io.PrintWriter
import java.io.IOException
import org.apache.spark.{SparkConf,SparkContext}
import org.apache.spark.rdd.PairRDDFunctions
import scala.util.Random    
......
val data=sc.textFile(path)
    val kv=data.map{s=>
      val a=s.split(",")
      (a(0),a(1))
    }.cache()
    kv.first()
    val start=System.currentTimeMillis()
    for(tg<-target){
      kv.lookup(tg.toString)
    }

The error detail is :

value lookup is not a member of org.apache.spark.rdd.RDD[(String, String)]
[error]           kv.lookup(tg.toString)

What confused me is I have import import org.apache.spark.rdd.PairRDDFunctions, but it doesn't work . And when I run this in Spark-shell ,it runs well.

maasg · Accepted Answer · 2015-05-20 07:45:55Z

1

import org.apache.spark.SparkContext._ to have access to the implicits that let you use PairRDDFunctions on a RDD of type (K,V). There's no need to directly import PairRDDFunctions

answered May 20, 2015 at 7:45

maasg

37.5k14 gold badges91 silver badges117 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

zhang Over a year ago

Thanks for you help.But I met new problem like this :[error] in package org.apache.hadoop which is not available. [error] It may be completely missing from the current classpath, or the version on [error] the classpath might be incompatible with the version used when compiling SparkContext.class. [error] for(i<- 0 to totalnum-1){

The Archetypal Paul Over a year ago

That's a different question. Please create a new question for it. But it sounds like a classpath or version conflict, so check those carefully.

Collectives™ on Stack Overflow

value lookup is not a member of org.apache.spark.rdd.RDD[(String, String)]

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related