Spark Scala Datasets using Java Classes

Question

I am creating a Spark application using the Scala binding. But some of my model's (classes) are written in Java. When I try to create a Dataset based on Scala Case Class, it works fine and all the columns are visible when I do show(). But when I create a Dataset based on a Java Class all the columns are packed in a single column named value.

Scala Case Class Example:

case class Person(name: String, age: Int)

Execution:

sqlContext.createDataset(Seq(Person("abcd", 10))).show()

Output:

name | age

abcd | 10

Java Class Example:

class Person {
  public String name;
  public int age;
  public Person (String name, int age) {
    this.name = name;
    this.age = age;
  }
}

Execution:

sqlContext.createDataset(Seq(Person("abcd", 10))).show()

Output:

value

[01 00 63 6F 6D 2...]

Are we not suppose to use Java classes as models with Spark Scala app? How do we resolve this issue?

OldWolfs · Accepted Answer · 2019-08-01 12:02:43Z

You can use Java classes to create Datasets but you need to explictly define bean for that class (works like that in java). In addition you need to define getter/setter methods to define bean and your class definition should have public keyword(spark complains about some compliation errors). Hope it works okay for you.

Class

public class Person {
  private String name;
  private int age;

  public Person (String name, int age) {
    this.name = name;
    this.age = age;
  }

  public String getName() {
    return name;
  }

  public void setName(String name) {
    this.name = name;
  }

  public int getAge() {
    return age;
  }

  public void setAge(int age) {
    this.age = age;
  }
}

Execution

implicit val personEncoder = Encoders.bean(classOf[Person])
sql.createDataset(Seq(new Person("abcd", 10))).show()

Result

+---+----+
|age|name|
+---+----+
| 10|abcd|
+---+----+

Collectives™ on Stack Overflow

Spark Scala Datasets using Java Classes

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related