I've been hacking at this problem since 2 months ago! Im tired, confused, burnt out and disillusioned !
I want the code below to read a file when its dropped into a folder. Sometimes 2 or three files can be dropped into the folder in rapid succession. This will happen all day. I expect a total of 250000 files a year to be dropped into the folder. I wish the contents of the file to be entered into a database. Each file contains one line of text. I split the text into four parts and insert into the database.
An example of a txt file being sftp'ed into the folder looks like this (single line of text). this file could be called textfile324324.txt for example
131115 004108 60 000890012
Two problems I have are that when a file is dropped into the folder, sometimes the SFTP script drops the file into the folder its still writing the file as filesystemwatcher is reading it, so I end up with a TMP file in the folder and not a txt file.
Also in testing, if I manually drop 1000 files in the folder in one go (lets go mad!), only around 700 files are read (and contents entered into database with no issues) but then the remaining files are not read. I dont see any errors at this point. The script just stops bothering to read files. I wonder if this if buffer size?
So Ive tried and tried since Decemeber to optimise the code below, I have various versions of it, and each time I hit a brick wall.
Where am I going wrong? Please note that I am new to C# and I dont fully understand threading. If I do need to be threading and offloading tasks could someone point me in the right direction with an example script?
I appreciate any help anyone is willing to give! Ive posted here before but I absolutely need a bit of hand holding with this one. Perhaps someone has coded similar before?
using System;
using System.IO;
using System.Data.SqlClient;
namespace IDGFileSystemWatcher
{
class Program
{
static void Main(string[] args)
{
string path = "C:/Test/";
MonitorDirectory(path);
Console.ReadKey();
}
private static void MonitorDirectory(string path)
{
FileSystemWatcher fileSystemWatcher = new FileSystemWatcher();
fileSystemWatcher.Path = path;
fileSystemWatcher.Created += FileSystemWatcher_Created;
fileSystemWatcher.EnableRaisingEvents = true;
}
private static void FileSystemWatcher_Created(object sender, FileSystemEventArgs e)
{
string txtContents;
string dir = "C:/Test/";
string textFile = dir + e.Name;
//string text = File.ReadAllText(textFile);
FileStream fs = new FileStream(textFile, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);
StreamReader sr = new StreamReader(fs);
txtContents = sr.ReadToEnd();
sr.Close();
string[] splitstring = txtContents.Split(new char[0], StringSplitOptions.RemoveEmptyEntries);
DateTime now = DateTime.Now;
SqlConnection connection = new SqlConnection();
connection.ConnectionString =
"Data Source=computername;" +
"Initial Catalog=contents;" +
"User id=userid;" +
"Password=password;";
string query = "INSERT INTO dbo.database (Field1, Field2, Field3, Field4, TStamp) VALUES (@Field1, @Field2, @Field3, @Field4, @TStamp)";
using (SqlCommand command = new SqlCommand(query, connection))
{
command.Parameters.AddWithValue("@Field1", splitstring[0]);
command.Parameters.AddWithValue("@Field2", splitstring[1]);
command.Parameters.AddWithValue("@Field3", splitstring[2]);
command.Parameters.AddWithValue("@Field4", splitstring[3]);
command.Parameters.AddWithValue("@TStamp", now);
connection.Open();
int result = command.ExecuteNonQuery();
// Check Error
if (result < 0)
{
Console.WriteLine("Error inserting data into database!");
}
// connection.Close();
}
}
}
}
I was expecting the script to throw an error for whatever reason it stops reading files.
SqlConnectionin ausingstatement. It is necessary to return the connection to the pool as soon as possible.