Encoder[Row] in Scala Spark

Question

I'm trying to perform a simple map on a Dataset[Row] (DataFrame) in Spark 2.0.0. Something as simple as this

val df: DataSet[Row] = ...
df.map { r: Row => r }

But the compiler is complaining that I'm not providing the implicit Encoder[Row] argument to the map function:

not enough arguments for method map: (implicit evidence$7: Encoder[Row]).

Everything works fine if I convert to an RDD first ds.rdd.map { r: Row => r } but shouldn't there be an easy way to get an Encoder[Row] like there is for tuple types Encoders.product[(Int, Double)]?

[Note that my Row is dynamically sized in such a way that it can't easily be converted into a strongly-typed Dataset.]

Dan · Accepted Answer · 2020-08-12 21:18:55Z

3

SSry to be a "bit" late. Hopefully this helps to someone who is hitting the problem right now. Easiest way to define encoder is deriving the structure from existing DataFrame:

val df = Seq((1, "a"), (2, "b"), (3, "c").toDF("id", "name")
val myEncoder = RowEndocer(df.schema)

Such approach could be useful when you need altering existing fields from your original DataFrame.

If you're dealing with completely new structure, explicit definition relying on StructType and StructField (as suggested in @Reactormonk 's little cryptic response).

Example defining the same encoder:

val myEncoder2 = RowEncoder(StructType(
  Seq(StructField("id", IntegerType), 
      StructField("name", StringType)
  )))

Please remember org.apache.spark.sql._, org.apache.spark.sql.types._ and org.apache.spark.sql.catalyst.encoders.RowEncoder libraries have to be imported.

answered Aug 12, 2020 at 21:18

Dan

5244 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Daniel Darabos Over a year ago

Instead of RowEncoder(df.schema) you can simply use df.encoder.

Daniel Darabos · Accepted Answer · 2021-11-11 08:44:40Z

2

In your specific case where the mapped function does not change the schema, you can pass in the encoder of the DataFrame itself:

df.map(r => r)(df.encoder)

answered Nov 11, 2021 at 8:44

Daniel Darabos

27.6k10 gold badges108 silver badges122 bronze badges

Comments

Reactormonk · Accepted Answer · 2016-10-01 08:49:44Z

1

An Encoder needs to know how to pack the elements inside the Row. So you could write your own Encoder[Row] by using row.structType which determines the elements of your Row at runtime and uses the corresponding decoders.

Or if you know more about the data that goes into Row, you could use https://github.com/adelbertc/frameless/

answered Oct 1, 2016 at 8:49

Reactormonk

21.8k16 gold badges78 silver badges128 bronze badges

Collectives™ on Stack Overflow

Encoder[Row] in Scala Spark

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related