0

What I want

To be able to insert multiple rows into a table, in SQL Server, using python, and have my INSTEAD OF INSERT trigger only triggered when all rows have been inserted.

Background

I have an INSTEAD OF INSERT trigger on my table which performs actions on previously unseen data.

The usual mode of operation is that periodically, multiple rows are inserted at a time.

When I'm inserting from python (using pyodbc), if I use:

cursor.executemany('INSERT INTO table_name VALUE (?, ?, ...)', list_of_lists)

The trigger will be triggered for every row - not what I want.

I'm currently using:

sql = 'INSERT INTO table_name VALUES (...),(...)...;'  # string built with format-strings
cursor.execute(sql)

This theoretically opens me up to SQL injection, but my environment is locked down so not really an issue.
Also, I don't know what the max limit for the query string is. I've tried to find the limit out using the method in this post, but I get back an empty result. So I'm not sure if there is no limit or if it is unknown.

The number of rows I'd be inserting at a time isn't expected to be crazy (no more than 1-2k rows, handful of columns). So I don't imagine I'd hit a limit but I don't want to be surprised later.

There is then the option of BULK INSERT with the FIRE_TRIGGERS option, but that would require uploading my data as a file to the machine running SQL Server. This is complexity that I would like to avoid.

Is there a better option or is my current solution fine?

3
  • Have you considered using a table type parameter? Commented May 27, 2021 at 14:48
  • @Larnu no I haven't. My SQL-fu is pretty weak. What would that look like? Commented May 27, 2021 at 14:56
  • @DavidBrowne-Microsoft not true. I demonstrate using one in my answer, and the on the answer I cite there is another answer confirming the version support was added. Commented May 27, 2021 at 15:16

1 Answer 1

2

Seems like you could use a Table Type Parameter to do this. I will note that my Python is not great, so I don't know if you can do this without using a Stored Procedure.

Anyway, firstly let's create a sample table:

CREATE TABLE dbo.SomeTable (ID int IDENTITY(1,1),
                            SomeString varchar(10),
                            SomeInt int,
                            SomeDate date,
                            SomeGUID uniqueidentifier);
GO

And then a sample type. I am intentionally omitting SomeGUID:

CREATE TYPE dbo.SomeType AS table (SomeString varchar(10),
                                   SomeInt int,
                                   SomeDate date);

You'll then need to create a procedure that accepts the above TYPE as a parameter. In the below I've defined a static value for the entire data set for SomeGUID to demonstrate that all the rows are inserted in a single call to the proc in the final result set:

CREATE OR ALTER PROC dbo.SomeProc @SomeData dbo.SomeType READONLY AS
BEGIN

    DECLARE @UID uniqueidentifier = NEWID();
    
    INSERT INTO dbo.SomeTable (SomeString, SomeInt, SomeDate, SomeGUID)
    SELECT SomeString,
           SomeInt,
           SomeDate,
           @UID
    FROM @SomeData;
END;
GO

Now the python. For this I wrote the simple script below, referencing this answer for some help. I've included the whole script for a full working solution, though obviously some won't be applicable to your script and you might not be using an environmental file:

#!/usr/bin/env python3
#Import the bare minimums for this to work
import pyodbc, os
from dotenv import load_dotenv

#Get the details from my environemental file
load_dotenv()
SQLServer = os.getenv('SQL_SERVER')
SQLDatabase = os.getenv('SQL_DATABASE')
SQLLogin = os.getenv('SQL_LOGIN')
SQLPassword = os.getenv('SQL_PASSWORD')

#Create the full connection string
SQLConnString = 'Driver={ODBC Driver 17 for SQL Server};Server=' + SQLServer + ';Database=' + SQLDatabase + ';UID='+ SQLLogin +';PWD=' + SQLPassword

#Define the data that is going to be inserted into the table.
MyData = [('abc',1,'20200527'),('def',2,'20210527')]
#Turn it into a tuple. explained in https://stackoverflow.com/a/61156422/2029983
params = (MyData,) 

#And then execute the procedure, passing params as the parameters; which is a table
with pyodbc.connect(SQLConnString,timeout=20) as sqlConn:
    with sqlConn.cursor() as cursor:
        cursor.execute('EXEC dbo.SomeProc ?;', params)
        sqlConn.commit()

And then, when I run a SELECT * FROM dbo.SomeTable I get the below data:

ID SomeString SomeInt SomeDate SomeGUID
1 abc 1 2020-05-27 945a3656-54ee-47a2-962c-7a34c7f50635
2 def 2 2021-05-27 945a3656-54ee-47a2-962c-7a34c7f50635
Sign up to request clarification or add additional context in comments.

6 Comments

The SELECT query should work without a stored procedure. You're just sending a TSQL batch with a parameter. Still must create the Table Type, though.
I've personally, however, got no idea how you define the data to into the TYPE with just s SELECT @DavidBrowne-Microsoft .cursor.execute('INSERT INTO dbo.SomeTable(SomeString,SomeInt,SomeDate) SELECT * FROM ?;', params) for example, does not work.
Interesting. In .NET you must specify the table type in the client's SqlParameter. Perhaps pyodbc is looking at the stored procedure parameters to determine the table type.
That's what I would assumed, @DavidBrowne-Microsoft . I haven't run a trace on the above the confirm; I might later if my curiosity gets the better of me.
Yep, that it exactly what is does, @DavidBrowne-Microsoft .
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.