How to add a new nullable String column in a DataFrame using Scala

Question

There are probably at least 10 question very similar to this, but I still have not found a clear answer.

How can I add a nullable string column to a DataFrame using scala? I was able to add a column with null values, but the DataType shows null

val testDF = myDF.withColumn("newcolumn", when(col("UID") =!= "not", null).otherwise(null))

However, the schema shows

root
 |-- UID: string (nullable = true)
 |-- IsPartnerInd: string (nullable = true)
 |-- newcolumn: null (nullable = true)

I want the new column to be string |-- newcolumn: string (nullable = true)

Please don't mark as duplicate, unless it's really the same question and in scala.

Try myDF.withColumn("newcolumn", lit(null).cast("string")). — Leo C
– Leo C, Commented Oct 17, 2019 at 23:06

Kombajn zbożowy · Accepted Answer · 2019-10-17 22:54:26Z

3

Just explicitly cast null literal to StringType.

scala> val testDF = myDF.withColumn("newcolumn", when(col("UID") =!= "not", lit(null).cast(StringType)).otherwise(lit(null).cast(StringType)))

scala> testDF.printSchema

root
 |-- UID: string (nullable = true)
 |-- newcolumn: string (nullable = true)

answered Oct 17, 2019 at 22:54

Kombajn zbożowy

10.8k5 gold badges33 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Raphael Roth · Accepted Answer · 2019-10-18 08:30:43Z

1

Why do you want a column which is always null? There are several ways, I would prefer the solution with typedLit:

myDF.withColumn("newcolumn", typedLit[String](null))

or for older Spark versions:

myDF.withColumn("newcolumn",lit(null).cast(StringType))

answered Oct 18, 2019 at 8:30

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Collectives™ on Stack Overflow

How to add a new nullable String column in a DataFrame using Scala

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related