DataFrame Initialization with null values

Question

I am trying to create a dataframe with One row whose values are null.

val df = Seq(null,null).toDF("a","b")

Faced issues even if we used null.instanceof also with no success.

val df = Seq(null.asInstanceOf[Integer],null.asInstanceOf[Integer]).toDF("a","b")

This works but I don't like to specify the type of field mostly it should be string.

Tzach Zohar · Accepted Answer · 2018-08-28 15:05:36Z

3

I'm assuming you want a two-column DF, in that case each entry should be a tuple or a case-class. If that's the case, you can also explicitly state the type of the Seq so that you don't have use asInstanceOf:

val df = Seq[(Integer, Integer)]((null, null)).toDF("a","b")

answered Aug 28, 2018 at 15:05

Tzach Zohar

37.9k3 gold badges83 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

loneStar Over a year ago

Can we get nulltype which we get if we use withColumn("a",lit(null)) import org.apache.spark.sql.types.NullType. I mean right know schema is integer and integer. I am loking for null, null df.withColumn("test", lit(null)).printSchema root |-- a: integer (nullable = true) |-- b: integer (nullable = true) |-- test: null (nullable = true)

Tzach Zohar Over a year ago

How is NullType useful? Anyway, I think the best way to achieve that is using an explicit schema, as suggested by @MansoorBabaShaik

loneStar Over a year ago

NullType makes more sense than String or Integer types. I think that the reason behind the NullType when we use the lit

Raphael Roth · Accepted Answer · 2018-08-28 17:28:42Z

2

My preferred way is to use Option.empty[A]:

val df = Seq((Option.empty[Int],Option.empty[Int])).toDF("a","b")

answered Aug 28, 2018 at 17:28

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Comments

pasha701 · Accepted Answer · 2018-08-28 14:38:22Z

0

Looks like missprint in "asInstanceOf", worked fine for me:

       List(null.asInstanceOf[Integer],null.asInstanceOf[Integer]).toDF("a").show(false)

answered Aug 28, 2018 at 14:38

pasha701

7,2171 gold badge17 silver badges22 bronze badges

Comments

Mansoor Baba Shaik · Accepted Answer · 2018-08-28 15:15:13Z

import org.apache.spark.rdd.RDD
import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
import org.apache.spark.sql.{DataFrame, Row, SparkSession}

object SparkApp extends App {

  val sparkSession: SparkSession = SparkSession.builder()
    .appName("Spark_Test_App")
    .master("local[2]")
    .getOrCreate()

  val schema: StructType = StructType(
    Array(
      StructField("a", IntegerType, nullable = true),
      StructField("b", IntegerType, nullable = true)
    )
  )

  import sparkSession.implicits._
  val nullRDD: RDD[Row] = Seq((null, null)).toDF("a", "b").rdd

  val df: DataFrame = sparkSession.createDataFrame(nullRDD, schema)

  df.printSchema()

  df.show()

  sparkSession.stop()
}

Collectives™ on Stack Overflow

DataFrame Initialization with null values

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related