1

I have this powershell script that would work if my DEST table ONLY had the columns listed in the select from my SOURCE server, but the DEST table has more. I haven't been able to find anything that gives examples on how to specify the columns from my dest table I want to insert into. Note that the SourceServer and DestServer are not linked servers.

Param ( 
  #[parameter(Mandatory = $true)] 
  [string] $SrcServer = "SourceServer", 
  [parameter(Mandatory = $true)] 
  [string] $SrcDatabase = "SourceDb", 
  #[parameter(Mandatory = $true)] 
  [string] $SrcTable = "stage.InternalNotes",
  #[parameter(Mandatory = $true)] 
  [string] $DestServer = "DestServer", 
  #[parameter(Mandatory = $true)] 
  [string] $DestDatabase = "DestDb", 
  [parameter(Mandatory = $true)] 
  [string] $DestTable = "dbo.InternalNotes", 
)

Function ConnectionString([string] $ServerName, [string] $DbName)
{
  "Data Source=$ServerName;Initial Catalog=$DbName;Integrated Security=True;User ID=$UID;Password=$PWD;"
}

$SrcConnStr = ConnectionString $SrcServer $SrcDatabase 
$SrcConn = New-Object System.Data.SqlClient.SQLConnection($SrcConnStr) 
$CmdText = "SELECT
   ino.UserId
  ,ino.StoreId
  ,ino.PostedById
  ,ino.DatePosted
  ,ino.NoteSubject
  ,ino.NoteText
  ,ino.NoteType
  ,ino.Classify
  ,ino.CreatedBy
  ,ino.CreatedUtc
  ,IsReadOnly = 0
FROM
   stage.InternalNotes AS ino
"  
$SqlCommand = New-Object system.Data.SqlClient.SqlCommand($CmdText, $SrcConn) 
$SrcConn.Open() 
[System.Data.SqlClient.SqlDataReader] $SqlReader = $SqlCommand.ExecuteReader() 

Try 
{ 
  $DestConnStr = ConnectionString $DestServer $DestDatabase 
  $bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($DestConnStr, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity) 
  $bulkCopy.DestinationTableName = $DestTable 
  $bulkCopy.WriteToServer($sqlReader)
} 
Catch [System.Exception] 
{ 
  $ex = $_.Exception 
  Write-Host $ex.Message 
} 
Finally
{ 
  Write-Host "Table $SrcTable in $SrcDatabase database on $SrcServer has been copied to table $DestTable in $DestDatabase database on $DestServer" 
  $SqlReader.close() 
  $SrcConn.Close() 
  $SrcConn.Dispose() 
  $bulkCopy.Close() 
} 

Essentially, I need to be able to do this:

INSERT INTO dbo.InternalNotes --DEST Server table
    (
     userID
    ,StoreID
    ,PostedByID
    ,DatePosted
    ,NoteSubject
    ,NoteText
    ,NoteType
    ,Classify
    ,CreatedBy
    ,CreatedDateUTC
    ,IsReadOnly
    )
SELECT
    ino.UserId
   ,ino.StoreId
   ,ino.PostedById
   ,ino.DatePosted
   ,ino.NoteSubject
   ,ino.NoteText
   ,ino.NoteType
   ,ino.Classify
   ,ino.CreatedBy
   ,ino.CreatedUtc
   ,IsReadOnly = 0
FROM
    stage.InternalNotes AS ino --SOURCE Server table

Edits after getting everything to work based on the accepted answer:

For some reason it didn't like the line:

$bulkCopy = New-Object -TypeName Data.SqlClient.SqlBulkCopy -ArgumentList $DestSqlConnection, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity, $DestSqlTransaction;

It gave the error:

Cannot convert argument "1", with value: "[System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity", for "SqlBulkCopy" to type "System.Data.SqlClient.SqlBulkCopyOptions": "Cannot convert value "[System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity" to type "System.Data.SqlClient.SqlBulkCopyOptions". Error: "Unable to match the identifier name [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity to a valid enumerator name. Specify one of the following enumerator names and try again: Default, KeepIdentity, CheckConstraints, TableLock, KeepNulls, FireTriggers, UseInternalTransaction, AllowEncryptedValueModifications""

So Instead I changed it to this, and everything worked:

$bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($DestSqlConnection, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity,$DestSqlTransaction) 

1 Answer 1

1

To do manual column mapping, you need to populate SqlBulkCopy.ColumnMappings. If you don't specify the mapping, then as far as I know SqlBulkCopy will assume the first column in the select list or DataRow goes into the first ordinal column of the destination table.

For example:

$bulkCopy.DestinationTableName = $DestTable;
$bulkCopy.ColumnMappings.Add('sourceColumn1','destinationColumn1');
$bulkCopy.ColumnMappings.Add('sourceColumn2','destinationColumn2');
$bulkCopy.ColumnMappings.Add('sourceColumn3','destinationColumn3');
$bulkCopy.ColumnMappings.Add('sourceColumn4','destinationColumn4');
$bulkCopy.ColumnMappings.Add('sourceColumn5','destinationColumn5');

However, there's a number of other issues with your script.

Your connection string authentication section is nonsense:

`Integrated Security=True; User ID=$UID; Password=$PWD;`

Integrated Security=True says, "Use passthrough Windows authentication with currently logged on user." User ID=$UID; Password=$PWD; says, "Use SQL authentication with the specified username and password." You can't do both. You should specify only one or the other.

$SqlCommand = New-Object system.Data.SqlClient.SqlCommand($CmdText, $SrcConn)
[...]
$bulkCopy = New-Object Data.SqlClient.SqlBulkCopy($DestConnStr, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity) 

I may be wrong, but I'm pretty sure you're trying to pass two variables as one argument here. Just like with your ConnectionString function, I don't think you don't want parentheses here. In any case it's syntactically confusing. Do this instead:

$SqlCommand = New-Object -TypeName System.Data.SqlClient.SqlCommand -ArgumentList $CmdText, $SrcConn
[...]
$bulkCopy = New-Object -TypeName Data.SqlClient.SqlBulkCopy -ArgumentList $DestConnStr, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity

Speaking of that last one, I have another issue with it. SqlBulkCopy is powerful, but you really have to hold it's hand. By default, SqlBulkCopy doesn't run with any transaction benefits. That means that if it errors in the middle, well, too bad, your data has been partially updated. You can enable internal transactions, but then only the most recent batch of the inserts will be rolled back. You really need to manage your own transaction to get an all-or-nothing result.

So you'll end up with something like this:

Try { 
    $DestConnStr = ConnectionString $DestServer $DestDatabase

    # We have to open the connection before we can create the transaction
    $DestSqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $DestConnStr;
    $DestSqlConnection.Open();
    $DestSqlTransaction = $DestSqlConnection.BeginTransaction();
    $bulkCopy = New-Object -TypeName Data.SqlClient.SqlBulkCopy -ArgumentList $DestSqlConnection, [System.Data.SqlClient.SqlBulkCopyOptions]::KeepIdentity, $DestSqlTransaction;
    $bulkCopy.DestinationTableName = $DestTable
    $bulkCopy.ColumnMappings.Add('sourceColumn1','destinationColumn1');
    $bulkCopy.ColumnMappings.Add('sourceColumn2','destinationColumn2');
    $bulkCopy.ColumnMappings.Add('sourceColumn3','destinationColumn3');
    $bulkCopy.ColumnMappings.Add('sourceColumn4','destinationColumn4');
    $bulkCopy.ColumnMappings.Add('sourceColumn5','destinationColumn5');

    Try {
        $bulkCopy.WriteToServer($sqlReader)
        # Commit on success
        $DestSqlTransaction.Commit();
    }
    Catch {
        # Rollback on error
        $DestSqlTransaction.Rollback();
        # Rethrow the error to the outer catch block
        throw ($_);
    }
} 
Catch [System.Exception] { 
    $ex = $_.Exception 
    Write-Host $ex.Message 
} 
Finally { 
    [...]
}

I'd probably rewrite the above more because I don't like nested try blocks, but for a quick and dirty rewrite this will work. I don't think you'll run into any problems with distributed transaction problems doing this, but I may be wrong. I tend to use SSIS or linked servers when I need this sort of data pump.

Sign up to request clarification or add additional context in comments.

2 Comments

Thank you for you detailed answer! My only experience with Powershell is what I could find through my google searches. This script was mostly based off of blogs.technet.microsoft.com/heyscriptingguy/2011/05/06/… I have these types of projects set up through SSIS, but our servers are quite outdated (2012 I believe) and all of my projects are in 2015. Trying to find an easier way to do this than re-creating my packages in the lower version. I'll try your suggestions out next week!
With a couple tweaks, it's working like a charm! (will edit my question to add comments on what was changed) Thanks again!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.