Pyspark Dataframe select all columns with alias on few columns

Question

I have a dataframe which has a lot of columns (more than 50 columns) and want to select all the columns as they are with few column names renamed by maintaining the below order. I tried the following ,

cols = list(set(df.columns) - {'id','starttime','endtime'})
df.select(col("id").alias("eventid"),col("starttime").alias("eventstarttime"),col("endtime").alias("eventendtime"),*cols,lit(proceessing_time).alias("processingtime"))

and got the error , SyntaxError: only named arguments may follow *expression

Also, instead of *cols, i tried to pass a list of column type

df.select(col("id").alias("eventid"),col("starttime").alias("eventstarttime"),col("endtime").alias("eventendtime"),([col(x) for x in cols]),lit(proceessing_time).alias("processingtime"))

which gives the following error,

`TypeError: 'Column' object is not callable`

Any help is highly appreciated.

Suresh · Accepted Answer · 2018-08-25 08:51:35Z

7

We could append the columns together and select from df,

df.select([col("id").alias("eventid"),col("starttime").alias("eventstarttime"),col("endtime").alias("eventendtime")]+cols+[lit(proceessing_time).alias("processingtime")])

answered Aug 25, 2018 at 8:51

Suresh

5,8802 gold badges27 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark Dataframe select all columns with alias on few columns

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related