Scala Spark: Parse SQL string to Column

Question

I have two functions, foo and bar, that I want to write like follows:

def foo(df : DataFrame, conditionString : String) = 
  val conditionColumn : Column = something(conditionString) //help me define "something"
  bar(df, conditionColumn)
}
def bar(df : DataFrame, conditionColumn : Column) = {
  df.where(conditionColumn)
}

Where condition is a sql string like "person.age >= 18 AND person.citizen == true" or something.

Because reasons, I don't want to change the type signatures here. I feel this should work because if I could change the type signatures, I could just write:

def foobar(df : DataFrame, conditionString : String) = {
  df.where(conditionString)
}

As .where is happy to accept a sql string expression.

So, how can I turn a string representing a column expression into a column? If the expression were just the name of a single column in df I could just do col(colName), but that doesn't seem to take the range of expressions that .where does.

If you need more context for why I'm doing this, I'm working on a databricks notebook that can only accept string arguments (and needs to take a condition as an argument), which calls a library I want to take column-typed arguments.

werner · Accepted Answer · 2021-09-14 15:00:15Z

2

You can use functions.expr:

def expr(expr: String): Column

Parses the expression string into the column that it represents

answered Sep 14, 2021 at 15:00

werner

15k6 gold badges36 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scala Spark: Parse SQL string to Column

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related