Use string variable containing expression in selectExpr of dataframe

Question

I'm having problems trying to use a variable containing my expression in the selectExpr of a dataframe.

So I have my variable as:

expression = '"substr(value,1,1) as qffffffffbf3ef0cf","substr(value,2,1) as q6a0aaf20"'

And trying to use this on the dataframe as:

ascii_df.selectExpr(expression).show(1)

However I keep getting a mismatched input error. If I put the expression in directly as follows it works:

ascii_df.selectExpr("substr(value,1,1) as qffffffffbf3ef0cf","substr(value,2,1) as q6a0aaf20").show(1)

Is there a way of doing this in PySpark?

Shaido · Accepted Answer · 2018-09-06 03:55:48Z

2

You are in fact using two separate expressions. When using them directly in selectExpr you are using the expressions as two separate arguments to selectExpr:

selectExpr("substr(value,1,1) as qffffffffbf3ef0cf","substr(value,2,1) as q6a0aaf20")

However expression is a single string which is why it will not work. Instead change it to:

expressions = ["substr(value,1,1) as qffffffffbf3ef0cf","substr(value,2,1) as q6a0aaf20"]
ascii_df.selectExpr(expressions).show(1)

answered Sep 6, 2018 at 3:55

Shaido

28.6k26 gold badges76 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Use string variable containing expression in selectExpr of dataframe

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related