Count empty values in dataframe column in Spark (Scala)

Question

I'm trying to count empty values in column in DataFrame like this:

df.filter((df(colname) === null) || (df(colname) === "")).count()

In colname there is a name of the column. This works fine if column type is string but if column type is integer and there are some nulls this code always returns 0. Why is this so? How to change it to make it work?

In this thread you find more extensive answers. Just add .count(). stackoverflow.com/questions/39727742/… — Steffen Schmitz
– Steffen Schmitz, Commented Jun 2, 2017 at 14:13

Anahcolus · Accepted Answer · 2017-06-03 10:39:29Z

5

As mentioned on the question that df.filter((df(colname) === null) || (df(colname) === "")).count() works for String data types but the testing shows that null are not handled.

@Psidom's answer handles both null and empty but does not handle for NaN.

checking for .isNaN should handle all three cases

df.filter(df(colName).isNull || df(colName) === "" || df(colName).isNaN).count()

answered Jun 3, 2017 at 10:39

Anahcolus

42.1k6 gold badges75 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

akuiper · Accepted Answer · 2017-06-02 13:22:15Z

2

You can use isNull to test the null condition:

val df = Seq((Some("a"), Some(1)), (null, null), (Some(""), Some(2))).toDF("A", "B")
// df: org.apache.spark.sql.DataFrame = [A: string, B: int]

df.filter(df("A").isNull || df("A") === "").count
// res7: Long = 2

df.filter(df("B").isNull || df("B") === "").count
// res8: Long = 1

answered Jun 2, 2017 at 13:22

akuiper

216k33 gold badges363 silver badges380 bronze badges

Collectives™ on Stack Overflow

Count empty values in dataframe column in Spark (Scala)

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related