Apache spark Row getAs[String] : java.lang.Byte cannot be cast to java.lang.String

Question

I have a Spark Dataframe, which looks like this:

+-----------+-----+
|foo        |  bar|
+-----------+-----+
|          3|10119|
|          2| 4305|
+-----------+-----+

And it has the following schema

org.apache.spark.sql.types.StructType = StructType(
    StructField(foo,ByteType,true), 
    StructField(bar,LongType,false)
)

As you can see, the column foo is of ByteType.

I need to get the first row of foo as a String.

When I try

val fooStr = df.first.getAs[String](0)

I get cast exception :

java.lang.ClassCastException: java.lang.Byte cannot be cast to java.lang.String

However when I use toString, I am able to cast

val myStr = df.first.get(0).toString

Why is it that when I use Row.getAs[String] I get a casting exception , but when I use toString, there is no error. Is there any drawback to using toString?

werner · Accepted Answer · 2020-06-26 21:01:41Z

1

Row.getAs[T](i) is here defined as

def getAs[T](i: Int): T = get(i).asInstanceOf[T]

asInstanceOf[T] simply tries to cast the object to the desired type (see here) without any further transformations. If the type returned by get(i) and the desired type are not compatible (like Byte and String) a ClassCastException is thrown.

Calling toString on the return value of get(0) means however that Byte.toString() is called. This is not a cast but a regular method call that returns a String.

answered Jun 26, 2020 at 21:01

werner

15k6 gold badges36 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Apache spark Row getAs[String] : java.lang.Byte cannot be cast to java.lang.String

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related