snowflake-connector-python[pandas] write_pandas creates duplicate records in table

Question

I am attempting to copy data into snowflake on an AWS Lambda. I have a situation right now where I have a dataframe that has no duplicates in it. I verify this by checking my dataframe like so:

df.duplicated().any() and verify that it returns False

I then double check by filtering by what should be a unique value in the dataframe

df[df["myColumn"] == "uniqueValue"] and I get 1 result.

I then run the following:

write_pandas(
            conn=con,
            df=df,
            table_name=table_name,
            database=database,
            schema=schema,
            chunk_size=chunk_size,
            quote_identifiers=False,
        )

and then when the data lands in the Snowflake table and I query it, there are 5 of each row in the SF database.

I verified that this function only runs one time as well.

Why am I getting 5 duplicates?

EDIT OK so I realized it's not related to this package. The issue is that after 1 minute the lambda is triggered again, and then again 1 minute later, etc. until it's been triggered 5 times.

No idea why it's being triggered multiple times though because all of the executions succeed eventually, but there are 5 of them running before the first one actually completes

UPDATE

Verified that it's not a memory issue and not a timeout issue.

What I have noticed is that when an API Call is made to retrieve some external data is when the next lambda seems to be triggered. Not sure why that would play a role but it seems to be affecting it.

Also, it's not set at 5 times, it will just re-trigger every minute until the first lambda execution finishes. I can see that the logs stop when the API call starts, and it's at that same log mark that I see the next lambda execution start.

Jake Boomgaarden · Accepted Answer · 2024-04-08 11:37:26Z

1

I'm not sure if this is a Jenkins specific issue or not, but what I found is that I was invoking the function synchronously and after 1 minute, if the lambda had not responded, then it was triggering it again... running with the invoke-async cli option instead of invoke lead to the duplication stopping.

answered Apr 8, 2024 at 11:37

Jake Boomgaarden

3,6261 gold badge24 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

snowflake-connector-python[pandas] write_pandas creates duplicate records in table

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related