convert dataframe to list of rows pyspark glue

Question

How can I convert my dataframe df to a list of rows?

Code

df = glueContext.create_dynamic_frame_from_options(
    connection_type = "s3",
    connection_options = {"paths": ["s3://data/tmp1/file.csv"]},
    format = "csv",
)
df = df.toDF()
list = df.values.tolist()

Error

dataframe has no attribute values

do you mean df = df.toPandas() and then liste = df.values.tolist() ? — sygma
– sygma, Commented Jan 16, 2020 at 18:55
Isn't this a duplicate of stackoverflow.com/q/34817549/11301900 ? — AMC
– AMC, Commented Jan 16, 2020 at 22:53

E. Zeytinci · Accepted Answer · 2020-01-16 19:34:27Z

2

IMHO, you can use toPandas(),

df = glueContext.create_dynamic_frame_from_options(
    connection_type="s3", 
    connection_options={"paths": ["s3://data/tmp1/file.csv"]}, 
    format="csv")

df = df.toPandas()
liste = df.values.tolist()

edited Jan 16, 2020 at 19:34

answered Jan 16, 2020 at 19:10

E. Zeytinci

2,6332 gold badges23 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sandeep Fatangare · Accepted Answer · 2020-01-20 06:03:29Z

0

In glue, you may use DyanamicFrame.map() method (https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html#aws-glue-api-crawler-pyspark-extensions-dynamic-frame-map)

df.map(to_list)
def to_list(rec):
       rec["list"] = [rec["col1"], rec["col2"] ]
       del rec["col1"]
       del rec["col2"]

answered Jan 20, 2020 at 6:03

Sandeep Fatangare

2,15413 silver badges15 bronze badges

Collectives™ on Stack Overflow

convert dataframe to list of rows pyspark glue

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related