insert rows via a python generator using snowflake executemany function

Question

I have a snowflake table and I am using the Snowflake Python Connector to insert value into it. Since I have a lot of data I am using a python generator object to insert value into my table.

conn = snowflake.connector.connect(...)
with conn.cursor() as write_cur:
    write_cur.executemany("INSERT INTO TABLE_1 VALUES (%s, %s)",
                          generator_data)

but this seems to throw an error that generator has no attribute len() . Is there anyway I can use an iterator to insert values into a snowflake table.

It will be a big code, you can consider something like this for now ((x,y) for x,y in zip(random.sample(range(1, 10000000), 30000000),random.sample(range(1, 10000000), 30000000))) — Darkrider
– Darkrider, Commented Mar 4, 2022 at 6:00
executemany function expecting an iterable that has a length — Srinivas Reddy Thatiparthy
– Srinivas Reddy Thatiparthy, Commented Mar 4, 2022 at 7:38

Felipe Hoffa · Accepted Answer · 2022-03-04 07:21:36Z

1

If possible don't do this - individual inserts are slow.

Instead batch collect all the values to insert, and then in one INSERT bring in all the rows.

But let's talk about the code in the question: There's nothing to iterate through. For cursor() to iterate, first you need results out of somewhere to iterate through.

answered Mar 4, 2022 at 7:21

Felipe Hoffa

59.8k23 gold badges185 silver badges365 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

insert rows via a python generator using snowflake executemany function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related