Getting connection error while reading data from ElasticSearch using apache Spark & Scala

Question

I gave the following code

val conf = new org.apache.spark.SparkConf()
      .setMaster("local[*]")
      .setAppName("es-example")
      .set("es.nodes", "search-2meoihmu.us-est-1.es.amazonaws.com")

val sc = new org.apache.spark.SparkContext(conf)
val resource = "index/data"
val count = sc.esRDD(resource).count()
println(count)

using,

elastic search version=1.5.2
spark version=1.5.2
Scala version=2.10.4

and given library dependency as follows,

libraryDependencies += "org.elasticsearch" % "elasticsearch-spark_2.10" % "2.1.3"

I am getting following error while running the program

Exception in thread "main" org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed

How can I read data from elastic search using spark and Scala?

Did you figure this problem out? I am having the same problem, though I am on ES v1.7.5, Spark v1.6.0, ES-Spark connector v2.10-2.3.3, and Scala 2.10.6. — Jane Wayne
– Jane Wayne, Commented Nov 20, 2016 at 22:44

Jane Wayne · Accepted Answer · 2016-11-20 22:57:31Z

1

Please look at the option "es.nodes.wan.only". By default, the value for this key is set to "false", and when I set it to true, that exception went away. Here is the current documentation for the configuration values: https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html.

val conf = new org.apache.spark.SparkConf()
 .setMaster("local[*]")
 .setAppName("es-example")
 .set("es.nodes", "search-2meoihmu.us-est-1.es.amazonaws.com")
 .set("es.nodes.wan.only", "true")

Note that the doc specifies to flip this value to true for environments like those on AWS, but this exception happened for me when attempting to point to a VM with Elasticsearch running.

answered Nov 20, 2016 at 22:57

Jane Wayne

8,97523 gold badges83 silver badges137 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ruxi Zhang Over a year ago

How would you make this connection between elasticsearch and spark when they sit on different aws cluster, which requires aws authentication?

Collectives™ on Stack Overflow

Getting connection error while reading data from ElasticSearch using apache Spark & Scala

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related