“value $ is not a member of StringContext” - Missing Scala plugin?

Question

I'm using maven with scala archetype. I'm getting that error:

“value $ is not a member of StringContext”

I already tried to add several things in pom.xml, but nothing worked very well...

My code:

import org.apache.spark.ml.evaluation.RegressionEvaluator
import org.apache.spark.ml.regression.LinearRegression
import org.apache.spark.ml.tuning.{ParamGridBuilder, TrainValidationSplit}
// To see less warnings
import org.apache.log4j._
Logger.getLogger("org").setLevel(Level.ERROR)


// Start a simple Spark Session
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().getOrCreate()

// Prepare training and test data.
val data = spark.read.option("header","true").option("inferSchema","true").format("csv").load("USA_Housing.csv")

// Check out the Data
data.printSchema()

// See an example of what the data looks like
// by printing out a Row
val colnames = data.columns
val firstrow = data.head(1)(0)
println("\n")
println("Example Data Row")
for(ind <- Range(1,colnames.length)){
  println(colnames(ind))
  println(firstrow(ind))
  println("\n")
}

////////////////////////////////////////////////////
//// Setting Up DataFrame for Machine Learning ////
//////////////////////////////////////////////////

// A few things we need to do before Spark can accept the data!
// It needs to be in the form of two columns
// ("label","features")

// This will allow us to join multiple feature columns
// into a single column of an array of feautre values
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.linalg.Vectors

// Rename Price to label column for naming convention.
// Grab only numerical columns from the data
val df = data.select(data("Price").as("label"),$"Avg Area Income",$"Avg Area House Age",$"Avg Area Number of Rooms",$"Area Population")

// An assembler converts the input values to a vector
// A vector is what the ML algorithm reads to train a model

// Set the input columns from which we are supposed to read the values
// Set the name of the column where the vector will be stored
val assembler = new VectorAssembler().setInputCols(Array("Avg Area Income","Avg Area House Age","Avg Area Number of Rooms","Area Population")).setOutputCol("features")

// Use the assembler to transform our DataFrame to the two columns
val output = assembler.transform(df).select($"label",$"features")


// Create a Linear Regression Model object
val lr = new LinearRegression()

// Fit the model to the data

// Note: Later we will see why we should split
// the data first, but for now we will fit to all the data.
val lrModel = lr.fit(output)

// Print the coefficients and intercept for linear regression
println(s"Coefficients: ${lrModel.coefficients} Intercept: ${lrModel.intercept}")

// Summarize the model over the training set and print out some metrics!
// Explore this in the spark-shell for more methods to call
val trainingSummary = lrModel.summary

println(s"numIterations: ${trainingSummary.totalIterations}")
println(s"objectiveHistory: ${trainingSummary.objectiveHistory.toList}")

trainingSummary.residuals.show()

println(s"RMSE: ${trainingSummary.rootMeanSquaredError}")
println(s"MSE: ${trainingSummary.meanSquaredError}")
println(s"r2: ${trainingSummary.r2}")

and my pom.xml is that:

<project xmlns="http://maven.apache.org/POM/4.0.0" 

xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>test</groupId>
  <artifactId>outrotest</artifactId>
  <version>1.0-SNAPSHOT</version>
  <name>${project.artifactId}</name>
  <description>My wonderfull scala app</description>
  <inceptionYear>2015</inceptionYear>
  <licenses>
    <license>
      <name>My License</name>
      <url>http://....</url>
      <distribution>repo</distribution>
    </license>
  </licenses>

  <properties>
    <maven.compiler.source>1.6</maven.compiler.source>
    <maven.compiler.target>1.6</maven.compiler.target>
    <encoding>UTF-8</encoding>
    <scala.version>2.11.5</scala.version>
    <scala.compat.version>2.11</scala.compat.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-mllib_2.11</artifactId>
      <version>2.0.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-core_2.11</artifactId>
      <version>2.0.1</version>
    </dependency>
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>2.0.2</version>
    </dependency>
    <dependency>
      <groupId>com.databricks</groupId>
      <artifactId>spark-csv_2.11</artifactId>
      <version>1.5.0</version>
    </dependency>

    <!-- Test -->
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.specs2</groupId>
      <artifactId>specs2-junit_${scala.compat.version}</artifactId>
      <version>2.4.16</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.specs2</groupId>
      <artifactId>specs2-core_${scala.compat.version}</artifactId>
      <version>2.4.16</version>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.scalatest</groupId>
      <artifactId>scalatest_${scala.compat.version}</artifactId>
      <version>2.2.4</version>
      <scope>test</scope>
    </dependency>
  </dependencies>

  <build>
    <sourceDirectory>src/main/scala</sourceDirectory>
    <testSourceDirectory>src/test/scala</testSourceDirectory>
    <plugins>
      <plugin>
        <!-- see http://davidb.github.com/scala-maven-plugin -->
        <groupId>net.alchim31.maven</groupId>
        <artifactId>scala-maven-plugin</artifactId>
        <version>3.2.0</version>
        <executions>
          <execution>
            <goals>
              <goal>compile</goal>
              <goal>testCompile</goal>
            </goals>
            <configuration>
              <args>
                <!--<arg>-make:transitive</arg>-->
                <arg>-dependencyfile</arg>
                <arg>${project.build.directory}/.scala_dependencies</arg>
              </args>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.18.1</version>
        <configuration>
          <useFile>false</useFile>
          <disableXmlReport>true</disableXmlReport>
          <!-- If you have classpath issue like NoDefClassError,... -->
          <!-- useManifestOnlyJar>false</useManifestOnlyJar -->
          <includes>
            <include>**/*Test.*</include>
            <include>**/*Suite.*</include>
          </includes>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

I have no idea about how to fix it. Does anybody have any idea?

yes, but it doesn't work. It continues with the same error: “value $ is not a member of StringContext” — Thaise
– Thaise, Commented May 26, 2017 at 20:54
You'll need to remove the spark-csv from your pom.xml as this will cause an runtime error — eliasah
– eliasah, Commented May 26, 2017 at 21:11
I had to import spark.sqlContext.implicits._ in the same sub function where i was using $, just importing implicits in main() did not work. — Sudheer Palyam
– Sudheer Palyam, Commented Mar 6, 2018 at 5:39

Jacek Laskowski · Accepted Answer · 2017-05-27 12:59:15Z

116

Add this.. it will work

val spark = SparkSession.builder().getOrCreate()    
import spark.implicits._ // << add this

edited May 27, 2017 at 12:59

Jacek Laskowski

75k28 gold badges253 silver badges440 bronze badges

answered May 26, 2017 at 21:11

Apurva Singh

5,0785 gold badges39 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Haroun Mohammedi · Accepted Answer · 2017-05-26 23:40:45Z

57

You can use the col function instead just import it like this :

import org.apache.spark.sql.functions.col

And then change the $"column" to col("column")

Hope it helps

answered May 26, 2017 at 23:40

Haroun Mohammedi

2,44414 silver badges28 bronze badges

1 Comment

John Haberstroh Over a year ago

This worked better for me since my spark object is not in scope for the function call I need to make, since I am writing functions that act on for dataFrame objects.

y2k-shubham · Accepted Answer · 2018-08-21 14:52:04Z

7

@Apurva's answer initially worked for me in that the error vanished from IntelliJ
But then it resulted in "Could not find implicit value for spark" during sbt compile phase

I found a strange work-around by importing spark.implicits._ from SparkSession referenced from DataFrame instead of one obtained by getOrCreate

import df.sparkSession.implicits._

where df is a DataFrame

This could be because my code was placed inside a case class that received an implicit val spark: SparkSession parameter; but I'm not really sure as to why this fix worked for me

edited Aug 21, 2018 at 14:52

answered Nov 28, 2017 at 9:33

y2k-shubham

11.9k15 gold badges75 silver badges166 bronze badges

Comments

ni_i_ru_sama · Accepted Answer · 2019-03-20 15:55:24Z

0

I'm using spark 1.6. The above answers are great but unfortunately doesn't work in 1.6

The way I solved it was by using df.col("column-name")

val df = df_mid
         .withColumn("dt", date_format(df_mid.col("timestamp"), "yyyy-MM-dd"))
         .filter("dt != 'null'")

answered Mar 20, 2019 at 15:55

ni_i_ru_sama

3041 silver badge14 bronze badges

Collectives™ on Stack Overflow

“value $ is not a member of StringContext” - Missing Scala plugin?

4 Answers 4

Comments

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related