Spark BinaryType to Scala/Java

Question

I'm doing a spark app using scala with following data:

+----------+--------------------+
|        id|                data|
+----------+--------------------+
|    id1   |[AC ED 00 05 73 7...|
|    id2   |[CF 33 01 61 88 9...|
+----------+--------------------+

The schema shows:

root
 |-- id: string (nullable = true)
 |-- data: binary (nullable = true)

I tried to convert this dataframe into a map object, with id being key and data being value

I have tried:

df.as[(String, BinaryType)].collect.toMap

but I got following error:

java.lang.UnsupportedOperationException: No Encoder found for org.apache.spark.sql.types.BinaryType
- field (class: "org.apache.spark.sql.types.BinaryType", name: "_2")
- root class: "scala.Tuple2"

Should be Array[Byte].

user10938362
– user10938362

2020-03-11 00:45:15 +00:00
Commented Mar 11, 2020 at 0:45 — user10938362
– user10938362, Commented Mar 11, 2020 at 0:45

Sim · Accepted Answer · 2020-03-11 01:20:51Z

3

BinaryType is a Spark DataType. It maps in Scala/Java to Array[Byte].

Try df.as[(String, Array[Byte])].collect.toMap.

Make sure you've imported your sessions implicits, e.g., import spark.implicits._ so you gain the ability to create Encoder[T] instances implicitly.

answered Mar 11, 2020 at 1:20

Sim

13.6k11 gold badges69 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

JQ. Over a year ago

I tried your suggestion, the error is: result.as[(String, Array[Byte])].collect.toMap java.lang.IllegalArgumentException: Unsupported class file major version 57 at org.apache.xbean.asm6.ClassReader.<init>(ClassReader.java:166)

Sim Over a year ago

You are likely running the wrong Java version. See stackoverflow.com/questions/53583199/…

Prabhatika Vij Over a year ago

Source: spark.apache.org/docs/latest/api/java/org/apache/spark/sql/… (The data type representing Array[Byte] values. Please use the singleton DataTypes.BinaryType.)

Collectives™ on Stack Overflow

Spark BinaryType to Scala/Java

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related