how to select columns in pySpark dataframe from a variable in Python

Question

I have a pySpark dataframe in python as -

from pyspark.sql.functions import col
dataset = sqlContext.range(0, 100).select((col("id") % 3).alias("key"))

the column name is key and I would like to select this column using a variable.

myvar = "key"

now I want to select this column using the myvar variable in perhaps a select statement

I tried this

dataset.createOrReplaceTempView("dataset")
spark.sql(" select $myvar from dataset ").show

but it returns me an error

no viable alternative at input 'select $'(line 1, pos 8)

How do I achieve this in pySpark?

Note that I may have different columns in future and I want to pass more than 1 variables or perhaps a list into SELECT clause.

The only thing I can suggest you is to collect the data from dataframe and store in ur variable. — Manu Gupta
– Manu Gupta, Commented Sep 13, 2019 at 4:32
Just use select. You can pass in lists also. Read more here. — pault
– pault, Commented Sep 13, 2019 at 14:18

Daniel · Accepted Answer · 2019-09-13 08:51:46Z

4

dataset.select(myVar) will select a single column based on variable

.select can also take a list dataset.select([myVar, mySecondVar])

answered Sep 13, 2019 at 8:51

Daniel

1,2729 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jose Miguel · Accepted Answer · 2021-01-18 19:50:47Z

0

If your variable is a python list, you can also do this:

columns = ['column_a', 'column_b', 'column_c']

#select the list of columns
df_pyspark.select(*columns).show()

answered Jan 18, 2021 at 19:50

Jose Miguel

913 bronze badges