Spark - there is no type "Dataframe" in Java

Question

I am going through this and this . I clearly see the type"Dataframe" is defined for scala and not for Java. Does that mean type "Dataframe" is not technically supported but rather use Dataset[T] insetad ?

And also, from this , please share inputs about what does it mean - Does that signify the same fact Java doesnt have the construct for dataframe ?

Scala Dataset[T] & DataFrame (alias for Dataset[Row])

Java Dataset[T]

All Scala classes are visible to Java and vice versa. Have you added the spark and scala libraries to your Maven/Gradle/SBT? E.g. for Maven to your pom.xml file. — Danny Varod
– Danny Varod, Commented Jul 27, 2020 at 16:36
I did , i was referring to the javadoc and scaladoc and also the other question wrt DataFrame (alias for Dataset[Row]) - it is not mentioned in Java — Nag
– Nag, Commented Jul 27, 2020 at 16:41
@AlexeyRomanov I know, for Dataset[Row], forgot that java can't see these though. — Danny Varod
– Danny Varod, Commented Jul 27, 2020 at 17:48
@Nag why not use a scala class in your code? It will still compile into your jar, however, you will be able to use all the scala "goodies" such as lambdas, implicits, typetags, spark-encoders etc. — Danny Varod
– Danny Varod, Commented Jul 27, 2020 at 17:50

Levi Ramsey · Accepted Answer · 2020-07-27 17:13:30Z

1

Java has no concept of a type alias, so in Java, you would need to use

Dataset<org.apache.spark.sql.Row>

to refer to a Dataframe.

answered Jul 27, 2020 at 17:13

Levi Ramsey

20.8k1 gold badge20 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Nag Over a year ago

as simple as that. thanks . so, no "DataFrame" construct in java

Collectives™ on Stack Overflow

Spark - there is no type "Dataframe" in Java

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related