How can I pass a list of columns to select in pyspark dataframe?

Question

I have list column names.

columns = ['home','house','office','work']

and I would like to pass that list values as columns name in "select" dataframe.

I have tried it...

df_tables_full = df_tables_full.select('time_event','kind','schema','table',columns)

but I have received error below..

TypeError: Invalid argument, not a string or column: ['home', 'house', 'office',
'work'] of type <class 'list'>. For column literals, use 'lit', 'array', 'struct' 
or 'create_map' function.

Can you have any ideia? Thank you guys!

notNull · Accepted Answer · 2020-03-20 19:27:39Z

40

Use * before columns to unnest columns list and use in .select.

columns = ['home','house','office','work']

#select the list of columns
df_tables_full.select('time_event','kind','schema','table',*columns).show()

df_tables_full = df_tables_full.select('time_event','kind','schema','table',*columns)

answered Mar 20, 2020 at 19:27

notNull

31.8k4 gold badges41 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user2441441 Over a year ago

This only works from Python 3.5 onwards?

Collectives™ on Stack Overflow

How can I pass a list of columns to select in pyspark dataframe?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related