Add Multiple columns to dataframe scala

Question

I have a condition where I have to add 100 columns (to an existing DF)

The existing DF is like this

c1 c2 c3 c4

The 100 columns should be added after c2 so the output looks likes this

c1 c2 c5 c6 c7 c8 c9 ...... c100 c3 c4

I used the .withColumn to add the columns and arrange them in order using .select

Is there a better way to do this?

TIA

There is not enough information in question to answer. can you show us what do you want to achieve, what is this column contains, what have you tried so far? — Gaurang Shah
– Gaurang Shah, Commented Jul 27, 2021 at 18:59

kanielc · Accepted Answer · 2021-07-27 19:00:38Z

5

Use select. The withColumn approach gets less efficient with a lot of columns.

val midCols: Seq[Column] = ...
df.select(Seq(c1, c2) ++ midCols ++ Seq(c3, c4):_*)

That way the new version of the Dataset is created in a single transformation.

Good info for others reading this is that you can use that pattern to construct and add complex column expressions.

val colsToAdd: Seq[Column] = Seq('a * 2 as "next_a", split('b) as "b_arr")

Given the naming/alias is within the definitions already, when you select them, they will have the intended name. That way you don't have to deal with a rename after.

edited Jul 27, 2021 at 19:00

answered Jul 27, 2021 at 18:41

kanielc

1,3221 gold badge12 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Add Multiple columns to dataframe scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related