Spark Streaming: Write dataframe to ElasticSearch

Question

I using the following code to write a stream to elasticsearch from python (pyspark) application.

#Streaming code
query = df.writeStream \
.outputMode("append") \
.format("org.elasticsearch.spark.sql") \
.option("checkpointLocation", "/tmp/") \
.option("es.resource", "logs/raw") \
.option("es.nodes", "localhost") \
.start()

query.awaitTermination()

If I write the results to the console it works fine, also, if I write to ES - not in streaming mode, it works ok. This is the code I used to write to ES:

#Not streaming
df.write.format("org.elasticsearch.spark.sql") \
.mode('append') \
.option("es.resource", "log/raw") \
.option("es.nodes", "localhost").save("log/raw")

The thing is, I can't debug it, the code is running, but nothing is written to ES (in streaming mode).

Thanks,

What's the question then? What does not work?

Jacek Laskowski
– Jacek Laskowski

2018-03-05 18:19:49 +00:00
Commented Mar 5, 2018 at 18:19 — Jacek Laskowski
– Jacek Laskowski, Commented Mar 5, 2018 at 18:19

matlabit · Accepted Answer · 2018-04-01 19:08:07Z

3

Eventually did work out for me, the problem was technical (needed vpn)

query = df.writeStream \
.outputMode("append") \
.queryName("writing_to_es") \
.format("org.elasticsearch.spark.sql") \
.option("checkpointLocation", "/tmp/") \
.option("es.resource", "index/type") \
.option("es.nodes", "localhost") \
.start()

query.awaitTermination()

answered Apr 1, 2018 at 19:08

matlabit

8582 gold badges13 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

arctic_Oak · Accepted Answer · 2018-10-12 11:35:51Z

2

Code:

val stream = df
      .writeStream
      .option("checkpointLocation", checkPointDir)
      .format("es")
      .start("realtime/data")

SBT Dependency:

libraryDependencies += "org.elasticsearch" %% "elasticsearch-spark-20" % "6.2.4"

answered Oct 12, 2018 at 11:35

arctic_Oak

1,0442 gold badges14 silver badges29 bronze badges

Collectives™ on Stack Overflow

Spark Streaming: Write dataframe to ElasticSearch

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related