Spark Scala code does not compile when used with a encoder + by name

Question

This is a simplified version of my code that is failing to compile:

object CompileFail {
  import org.apache.spark.sql.{ Encoder, SparkSession }

  case class Foo(x: Int)

  def run(spark: SparkSession): Int => Int = {
    import spark.implicits._
    add(bar(encode[Foo]))
  }

  def bar(f: => Int): Int = 0

  def encode[A : Encoder]: Int = 0

  def add(a: Int)(b: Int) = a + b
}

It is failing with following non sensical message:

[error] Error while emitting CompileFail.scala
[error] value $u
[error] one error found
[error] Compilation failed

I am on Scala 2.12.15 and Spark 3.1.2 (but it fails on older Spark versions too).

Interesting to note:

If I change add(a)(b) to add(a, b) it compiles
If I change bar(f: => Int) to bar(f: Int) it compiles
If I change add(bar(encode[Foo])) to add(bar(encode[String])) it compiles

What am I doing wrong?

I strongly think it has something to do with the implicit's the Encoder library brings in. — Anant
– Anant, Commented Feb 21, 2022 at 19:03

Alfilercio · Accepted Answer · 2022-03-07 22:37:47Z

2

+100

I don't think you are doing anything wrong, it's a bug in scala resolution. Only doing this simple change works.

object CompileFail {
  import org.apache.spark.sql.{ Encoder, SparkSession }

  case class Foo(x: Int)

  def run(spark: SparkSession): Int => Int = {
    import spark.implicits._
    val i = encode[Foo]
    add(bar(i))
  }

  def bar(f: => Int): Int = 0

  def encode[A : Encoder]: Int = 0

  def add(a: Int)(b: Int) = a + b
}

It looks like it has conflicts with the implicit resolution and the lambda conversion of the function.

answered Mar 7, 2022 at 22:37

Alfilercio

1,1186 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark Scala code does not compile when used with a encoder + by name

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related