I'm creating a file transfer package that copies files into an output folder and then copies them into a separate archive. The files are frequently replaced with new files with a file name format such as xxxx20190427.zip I created a foreach loop in SSIS to copy / move the files but I can't figure out how to replace old versions. As I move the files into the output folder, I want to replace the file from the previous load. I'm trying to create a script task that creates a variable based on the substring of the variable declared in the foreach loop so that if the file xxxx20190427.zip is present, it checks for the "xxxx" portion in the filenames of my output folder and deletes that file if it's present so that only the newest file is available.
-
The following will get you all files in the requested format: Directory.GetFiles("your folder path", "xxxx*.zip");Metheny– Metheny2019-04-27 07:52:14 +00:00Commented Apr 27, 2019 at 7:52
-
I think that you are looking for the following answer: stackoverflow.com/questions/44591799/… . I didn't marked the question as duplicates since i am not sure if i understood what you are looking for exactly!!Hadi– Hadi2019-04-27 10:42:16 +00:00Commented Apr 27, 2019 at 10:42
-
What reason do you have for building this in SSIS?Nick.Mc– Nick.Mc2019-04-27 13:24:54 +00:00Commented Apr 27, 2019 at 13:24
-
@Nick.McDermaid it's going to be running as a step in my SSIS ETL package.jonsanchez311– jonsanchez3112019-04-28 03:13:13 +00:00Commented Apr 28, 2019 at 3:13
-
I don't quite understand what you're after but I do know that there are a bunch of proven standard data ingestion patterns that have been implemented and proven many times. Reading your explanation it sounds like you might have a complicated solution to what is really a simple pattern.If you could explain "big picture" what you are trying to do it will give context. If you could also add clear examples about your problem at hand, it's easier to understand than a large literary explanationNick.Mc– Nick.Mc2019-04-28 03:21:03 +00:00Commented Apr 28, 2019 at 3:21
|
Show 3 more comments
1 Answer
An overview of the process is below, with a couple C# Script Tasks used in this example. For the Script Tasks be sure to add the proper references as indicated below, with these being the using statements at the beginning of each section of code.
- Add an Execute SQL Task to the beginning of the Control Flow. The file names from the current execution will be held in a table that will be created in this task. Example SQL for this is below. This checks if the table exists, drops the table if it's present, and then creates it.
- You probably already have the file name for each iteration stored in a variable, but if not add a string variable at Index 0 of the Foreach Loop. This can be done on the Variable Mappings pane.
- Within the Foreach Loop add a Script Task after all the components. Add the string variable holding the file name in the
ReadOnlyVariablesfield. This script task (uses C#) will insert the file name used in each iteration to a table that will be used later. While doing the insert with C# is a slightly more involved than just an Execute SQL Task, theFileInfo.Nameproperty can be used to ensure only the file name is stored, as opposed to the file name with path. - After the Foreach Loop create another Script Task. If variables are used to hold the name of the output folder and file prefix add these in the
ReadOnlyVariablesfield as before. Code for this is also below. First the names of the files from the current execution are retrieved and stored in thecurrentExecutionFileslist. The files in the output directory are then enumerated through, with any files that have the prefix and were not used in the current package execution deleted. - This is optional, but I'd recommend adding another Execute SQL Task at the end of the package to drop the table holding the file names. The
DROPstatement from the first Execute SQL Task can just be used for this.
CREATE and DROP DDL for Execute SQL Task:
IF (OBJECT_ID(N'DATABASE.SCHEMA.FILES') IS NOT NULL)
BEGIN
DROP TABLE DATABASE.SCHEMA.FILES
END;
CREATE TABLE DATABASE.SCHEMA.FILES
(
[FILENAME] VARCHAR(250)
)
C# Script Task to Store File Names:
using System.Data;
using System.Data.SqlClient;
using System.IO;
string currentFile = Dts.Variables["User::FileNameVariable"].Value.ToString();
FileInfo currentFileInfo = new FileInfo(currentFile);
//Integrated Security for Windows authentication
string connectionStr = @"Data Source=Server;Initial Catalog=Database;Integrated Security=SSPI;";
string cmd = @"INSERT INTO DATABASE.SCHEMA.FILES ([FILENAME]) VALUES (@file)";
using (SqlConnection conn = new SqlConnection(connectionStr))
{
using (SqlCommand sql = new SqlCommand(cmd, conn))
{
SqlParameter pFileName = new SqlParameter("@file", SqlDbType.VarChar);
pFileName.Direction = ParameterDirection.Input;
pFileName.Size = 250;
//use FileInfo.Name to get only file path
pFileName.Value = currentFileInfo.Name;
sql.Parameters.Add(pFileName);
conn.Open();
//insert file name
sql.ExecuteNonQuery();
}
}
C# Script Task to Delete Old Files:
using System.Collections.Generic;
using System.Data.SqlClient;
using System.IO;
string outputFolder = Dts.Variables["User::OutputFolder"].Value.ToString();
string filePrefix = Dts.Variables["User::FilePrefix"].Value.ToString();
DirectoryInfo di = new DirectoryInfo(outputFolder);
List<string> currentExecutionFiles = new List<string>();
string connectionStr = @"Data Source=Server;Initial Catalog=Database;Integrated Security=SSPI;";
string cmd = @"SELECT [FILENAME] FROM DATABASE.SCHEMA.FILES";
using (SqlConnection conn = new SqlConnection(connectionStr))
{
using (SqlCommand sql = new SqlCommand(cmd, conn))
{
conn.Open();
using (SqlDataReader dr = sql.ExecuteReader())
{
if (dr.HasRows)
{
while (dr.Read())
{
//get file names from table holding names from current execution
currentExecutionFiles.Add(dr[0].ToString());
}
}
}
}
}
foreach (FileInfo fi in di.EnumerateFiles())
{
//check for file with prefix (case insensitive) that was not in current execution
if (fi.Name.StartsWith(filePrefix, StringComparison.InvariantCultureIgnoreCase)
&& !currentExecutionFiles.Contains(fi.Name))
{
fi.Delete();
}
}