Drop multiple columns from spark dataframe [duplicate]

Question

I'm trying to drop multiple columns from a dataframe. I've followed the example at https://stackoverflow.com/a/39818645/10836338 but it doesn't work for me. I am using spark 2.4.0

Here's what I'm doing right now. It does work, but I was wondering if there is a better way of doing this.

val colsToRemove = Seq("colA", "colB", "colC", etc) 

var filterdDF = df
 for(i <- 0 until colsToRemove.size){
    filterdDF = filterdDF.drop(colsToRemove(i))
}

drop also accepts a list of column name strings which can be used here. No need to loop. — Vamsi Prabhala
– Vamsi Prabhala, Commented Jan 3, 2019 at 19:26
I've already tried that but it returns an error. error: overloaded method value drop with alternatives: (col: org.apache.spark.sql.Column)org.apache.spark.sql.DataFrame <and> (colNames: String*)org.apache.spark.sql.DataFrame <and> (colName: String)org.apache.spark.sql.DataFrame — rasthiya
– rasthiya, Commented Jan 3, 2019 at 19:29

Fermat's Little Student · Accepted Answer · 2019-01-03 20:14:39Z

8

val filterdDF = df.drop(colsToRemove:_*)

answered Jan 3, 2019 at 20:14

Fermat's Little Student

6,3697 gold badges53 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Nikaido · Accepted Answer · 2020-04-09 14:16:52Z

-2

you can try this...

val colsToRemove = Seq("emp_id", "emp_city")

val filteredDF = ds1.select(ds1.columns.filter(colName => !colsToRemove.contains(colName)).map(colname => new Column(colname)).toList: _*)

edited Apr 9, 2020 at 14:16

Nikaido

4,6675 gold badges37 silver badges51 bronze badges

answered Jan 3, 2019 at 20:23

Andy Quiroz

8998 silver badges8 bronze badges

Collectives™ on Stack Overflow

Drop multiple columns from spark dataframe [duplicate]

2 Answers 2

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Linked

Related