Error while saving data to elasticsearch from spark - saveToEs

Question

I am trying to save the output of an rdd into elasticsearch. But when i try to send it, i face an error even after including few elasticsearch-spark libraries. I am new to elastic search and any help would be highly appreciated. Thanks.

import org.apache.spark.{SparkConf, SparkContext}
import org.elasticsearch.spark._

object ElasticSpark {

def main(args: Array[String]) {

val logfile = "/Users/folder/Desktop/logfile.rtf";
val conf = new SparkConf().setMaster("local[1]").setAppName("RddTest");   // set master can be given any cpu cores as local[*], spark clustr, mesos,
conf.set("es.index.auto.create", "true")
val sc = new SparkContext(conf);

val logdata = sc.textFile(logfile); // number of partitions
val NumA = logdata.filter(line=>line.contains("a")).count();
val wordcount = logdata.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey((a, b)=> a+ b);

println(wordcount.collect()); // doubt
wordcount.saveAsTextFile("/Users/folder/Desktop/sample") // success
wordcount.saveToEs("spark/docs")

}
}

Error

Error:(21, 15) value saveToEs is not a member of org.apache.spark.rdd.RDD[(String, Int)]
wordcount.saveToEs("spark/docs")
          ^
Error:(6, 12) object elasticsearch is not a member of package org
import org.elasticsearch.spark._
       ^

Vitalii Kotliarenko · Accepted Answer · 2016-04-24 07:56:10Z

2

ES support is not a part of Spark distribution, it's a part of elasticsearch-hadoop, so you need to provide this dependency. If you use Maven, add to your pom.xml:

<dependency>
  <groupId>org.elasticsearch</groupId>
  <artifactId>elasticsearch-hadoop</artifactId>
  <version>2.2.0</version>
</dependency>

For sbt, add to build.sbt:

libraryDependencies += "org.elasticsearch" % "elasticsearch-hadoop" % "2.2.0" % "compile"
resolvers ++= Seq("clojars" at "https://clojars.org/repo",
                  "conjars" at "http://conjars.org/repo",
                  "plugins" at "http://repo.spring.io/plugins-release",
                  "sonatype" at "http://oss.sonatype.org/content/groups/public/")

edited Apr 24, 2016 at 7:56

answered Apr 23, 2016 at 21:18

Vitalii Kotliarenko

2,96720 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Goutam Over a year ago

I get this error when i include it in sbt: Error:Error while importing SBT project:<br/>...<br/><pre>[error] unresolved dependency: clj-stacktrace#clj-stacktrace;0.2.2: not found [error] unresolved dependency: ring#ring-jetty-adapter;0.3.11: not found [error] unresolved dependency: ring#ring-servlet;0.3.11: not found [error] unresolved dependency: com.twitter#carbonite;1.4.0: not found [error] unresolved dependency: cascading#cascading-hadoop;2.6.3: not found [error] unresolved dependency: org.pentaho#pentaho-aggdesigner-algorithm;5.1.5-jhyde: not found

Vitalii Kotliarenko Over a year ago

You're right, additional resolvers have to be added to build.sbt. I've edited my answer. Btw, I did not find that fact mentioned somewhere in elastic documentation, so I just grab repositories from elasticsearch-hadoop Gradle project. It builds now, tested.

Collectives™ on Stack Overflow

Error while saving data to elasticsearch from spark - saveToEs

Error

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Error

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related