Retrieving a column value from a row object: no encoder found

Question

I wrote the following code to take some action based on a field value for each row:

import spark.implicits._ 
table.map(line =>
          column_names.map(column =>
                      if (line.getAs[Int](column)==0)
                          println("yes")
                      )
          )

However I encounter the following error:

java.lang.UnsupportedOperationException: No Encoder found for Unit
- array element class: "scala.runtime.BoxedUnit"
- root class: "scala.collection.immutable.List"

I understand that an encoder is required to convert objects and primitives in the Spark InternalRow, but the field that I'm trying to access contains an int, and based on my understanding the corresponding encoder should be available thanks to spark.implicits._.

Could you please tell me what am I missing and what should I do to fix it?

Raphael Roth · Accepted Answer · 2019-10-29 19:12:43Z

1

The problem is that your map-statement returns Unit (i.e. nothing), for which there is no encoder. If you just want to print, then use foreach instead:

table.foreach(line =>
  column_names.foreach(column =>
    if (line.getAs[Int](column)==0)
      println("yes")
  )
)

otherwise, return something (ensure that you also return the same type in the else-clause, e.g.

table.map(line =>
  column_names.map(column =>
    if (line.getAs[Int](column)==0) {
      println("yes")
      "yes"
     } else {
       "no"
     }
  )
)

answered Oct 29, 2019 at 19:12

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Retrieving a column value from a row object: no encoder found

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related