Snowflake Fetch Many in Python

Question

I'm trying to create a pandas data frame using the Snowflake Packages in python.

I run some query

sf_cur = get_sf_connector()
sf_cur.execute("USE WAREHOUSE Warehouse;")
sf_cur.execute("""select Query"""
)

print('done')

The output is roughly 21k rows. Then using

df = pd.DataFrame(sf_cur.fetchall())

takes forever, even on a limit sample of only 100 rows. Is there a way to optimize this, ideally the bigger query would be run in a loop so handling even bigger data sets would be ideal.

Pablo Ocaña · Accepted Answer · 2019-01-29 21:40:41Z

3

as fetchall() copies all the result in memory, you should try to iterate over the cursor object directly and map it to a data frame inside the for block

cursor.execute(query)
    for row in cursor:
    #build the data frame

Other example, just to show:

query = "Select ID from Users"
cursor.execute(query)
for row in cursor:
    list_ids.append(row["ID"])

edited Jan 29, 2019 at 21:40

answered Jan 25, 2019 at 3:39

Pablo Ocaña

573 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

JF4bes Over a year ago

Can you explain "build the dataframe" I'm not sure how to do this other than Fetch()?

Pablo Ocaña Over a year ago

@JeffreyForbes for example, let's say you have a list of User's ID's (just for showing the case): query = "Select ID from Users" cursor.execute(query) for row in cursor: list_ids.append(row["ID"])

Anmol Kankariya · Accepted Answer · 2020-05-11 13:50:31Z

0

Use df = cur.fetch_pandas_all() to build pandas dataframe on top of results.

answered May 11, 2020 at 13:50

Anmol Kankariya

2583 silver badges6 bronze badges

Collectives™ on Stack Overflow

Snowflake Fetch Many in Python

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related