0

I want to implement a generic encoder because I have a small project on Spark 1.6 and when I migrate it to the spark 2.0 its giving me the warning and error of 

Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.

at many places hence I want to implement a generic encoder and put it in package object. I just want to know how can I implement a generic Encoder in spark 2.0 ?

Improve this question
1

1 Answer 1

Reset to default
2

Developing custom encoders usually starts with composing existing encoders from org.apache.spark.sql.Encoders object.

Please note that many places in Spark check whether the encoder in use is an instance of ExpressionEncoder (i.e. SparkSession.createDataset) so you may want to review expression encoders as the base for custom development.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.