Using Filter Condition While Joining Spark Dataframes: Spark/Scala

Question

can someone please suggest me how to use filter while joining 2 dataframes in spark scala.I am trying below code.

    var name="abcd"
    var last_name="xyz"

    val df3 = df1.join(df2, df1("id") === df2("id"))
    .filter(df1("name")==='${name}').
    filter(df1("last_name")==='${last_name}')
    .drop(df1("name"))
    .drop(df2("name"))

But getting multiple error.

QuickSilver · Accepted Answer · 2020-05-18 15:35:41Z

2

Spark is not like java's JDBC APIs where we need wrap string with single quotes for where condition. Can you simple try using name variable w/o any quotes and $ sign

    var name="abcd"
    var last_name="xyz"
    val df3 = df1.join(df2, df1("id") === df2("id"))
    .filter(df1("name")===name && df1("last_name")===last_name)
    .drop(df1("name"))
    .drop(df2("name"))

edited May 18, 2020 at 15:35

answered May 18, 2020 at 15:17

QuickSilver

4,0452 gold badges15 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Up Ap Over a year ago

Yes it worked for adding multiple filter condition do i need to use multiple filter or i can use &&?

QuickSilver Over a year ago

please use && for multiple filters

Collectives™ on Stack Overflow

Using Filter Condition While Joining Spark Dataframes: Spark/Scala

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related