How to convert DataFrame to DynamicFrame object in Scala

Question

I'm trying to convert some of my pySpark code to Scala to improve performance. In AWS Glue (which uses Apache Spark) a script is automatically generated for you and it typically uses the DynamicFrame object to load, transform and write data out. However, the DynamicFrame class does not have all of the same functionalities as the DataFrame class and at times you have to convert back to a DataFrame object and vice versa to perform certain operations. Below is how I have converted from DataFrame to DynamicFrame objects in pySpark:

    // PySpark version
    // datasource is a DynamicFrame object
    datasource0 = datasource.toDF().limit(5000000)
    applymapping1 = DynamicFrame.fromDF(datasource0, glueContext, "applymapping1")

Is there an equivalent function to fromDF in Scala to revert back to a DynamicFrame object?

botchniaque · Accepted Answer · 2018-05-17 19:22:33Z

10

DynamicFrame object has an apply method:

val dynamicFrame = DynamicFrame(dataframe, glueContext)

answered May 17, 2018 at 19:22

botchniaque

5,1844 gold badges40 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

DrV Over a year ago

Thanks @botchniaque . This works and my code is up and running now. :)

Collectives™ on Stack Overflow

How to convert DataFrame to DynamicFrame object in Scala

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related