Spark app unable to write to elasticsearch cluster running in docker

Question

I have a elasticsearch docker image listening on 127.0.0.1:9200, I tested it using sense and kibana, It works fine, I am able to index and query documents. Now when I try to write to it from a spark App

val sparkConf = new SparkConf().setAppName("ES").setMaster("local")
sparkConf.set("es.index.auto.create", "true")
sparkConf.set("es.nodes", "127.0.0.1")
sparkConf.set("es.port", "9200")
sparkConf.set("es.resource", "spark/docs")


val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)
val numbers = Map("one" -> 1, "two" -> 2, "three" -> 3)
val airports = Map("arrival" -> "Otopeni", "SFO" -> "San Fran")
val rdd = sc.parallelize(Seq(numbers, airports))

rdd.saveToEs("spark/docs")

It fails to connect, and keeps on retrying

16/07/11 17:20:07 INFO HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Operation timed out 16/07/11 17:20:07 INFO HttpMethodDirector: Retrying request

I tried using IPAddress given by docker inspect for the elasticsearch image, that also does not work. However when I use a native installation of elasticsearch, the Spark App runs fine. Any ideas?

bp2010 · Accepted Answer · 2018-05-07 12:25:22Z

3

Also, set the config

es.nodes.wan.only to true

As mentioned in this answer if you are having issues writing to ES.

answered May 7, 2018 at 12:25

bp2010

2,50223 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Eyal.Dahari · Accepted Answer · 2016-07-17 21:38:38Z

1

Couple things I would check:

The Elasticsearch-Hadoop spark connector version you are working with. Make sure that it is not beta. There was a fixed bug related to the IP resolving.
Since 9200 is the default port, you may remove this line: sparkConf.set("es.port", "9200") and check.
Check that there is no proxy configured in your Spark environment or config files.
I assume that you run Elasticsaerch and Spark on the same machine. Can you try to configure your machine IP address instead of 127.0.0.1

Hope this helps! :)

answered Jul 17, 2016 at 21:38

Eyal.Dahari

7706 silver badges14 bronze badges

Comments

antike · Accepted Answer · 2020-02-05 13:31:37Z

1

Had the same problem and a further issue was that the confs set using sparkConf.set() didn't have an effect. But supplying the confs with the saving function worked, like this:

rdd.saveToEs("spark/docs", Map("es.nodes" -> "127.0.0.1", "es.nodes.wan.only" -> "true"))

answered Feb 5, 2020 at 13:31

antike

2312 silver badges6 bronze badges

Collectives™ on Stack Overflow

Spark app unable to write to elasticsearch cluster running in docker

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related