Scala - How to pass a string value to a data frame filter (Spark-Shell)

Question

I am using Spark 2.2.0 and Scala 2.11.8 in the Spark-Shell environment. I have a data frame df, and I need to filter out the previous day's data based on the value of column 'date' and then append the data to a HDFS location. (e.g. today is 2018-06-28, I need the data of 2018-06-27)

Below is the code:

 df.filter($"date" === "2018-06-27") .write.mode(SaveMode.Append).parquet("hdfs:/path..../date=2018-06-27")

I need the code above for automation, so I need to replace "2018-06-27" for the filter value as well as the directory name. So if I have a string -> date_test: String = 2018-06-27; The code below should be still working

 df.filter($"date" === "date_test") .write.mode(SaveMode.Append).parquet("hdfs:/path..../date=date_test")

How to do this?

What is the data type of your date column date or string

Chandan Ray
– Chandan Ray

2018-06-28 18:06:03 +00:00
Commented Jun 28, 2018 at 18:06 — Chandan Ray
– Chandan Ray, Commented Jun 28, 2018 at 18:06
The type of the date column is string

Allen
– Allen

2018-06-28 18:41:06 +00:00
Commented Jun 28, 2018 at 18:41 — Allen
– Allen, Commented Jun 28, 2018 at 18:41
why the downvote?

Jake
– Jake

2018-12-08 22:06:09 +00:00
Commented Dec 8, 2018 at 22:06 — Jake
– Jake, Commented Dec 8, 2018 at 22:06

Manoj Kumar Dhakad · Accepted Answer · 2018-06-28 19:30:37Z

2

You can apply filter conditions like below

//Input
+----------+
|      date|
+----------+
|2018-02-01|
|2017-01-02|
+----------+

//Solution: 
 val previousDate="'2018-02-01'"
 df.filter(s"date=$previousDate").show

//Output: 
+----------+
|      date|
+----------+
|2018-02-01|
+----------+

You can do like this for your solution

 val datetest:String="2018-02-01"
 df.filter(s"date='$datetest'").write.mode(SaveMode.Append).parquet(s"hdfs:/path..../$datetest")

edited Jun 28, 2018 at 19:30

answered Jun 28, 2018 at 18:58

Manoj Kumar Dhakad

1,9021 gold badge14 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Allen Over a year ago

So for automation purpose, It cannot contain the actually date '2018-02-01'. For example, if I have a string datetest: String = 2018-02-01. The code should be able to work like df.filter("date='datetest'").show

Manoj Kumar Dhakad Over a year ago

I modified answer you need to create your filter condition like datetest: String = "'2018-02-01'"

Allen Over a year ago

This works for the filter, but does it work for the directory name as well?

Manoj Kumar Dhakad Over a year ago

No, It will not work .Please see last part of answer It will work fine

Allen Over a year ago

Cool, I'll try it.

Collectives™ on Stack Overflow

Scala - How to pass a string value to a data frame filter (Spark-Shell)

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related