FileSystemWatcher InternalBufferSize limit is getting exceeded

Question

We are receiving files in FileSystemWatcher folder very frequently because of that InternalBufferSize limit is getting exceeded and files are being missed. I am using the max limit 65536 allowed for InternalBufferSize.

I have also gone through many questions related to this, but could not find appropriate solution.

My Concern is:

Can we optimize this code more accurately?

If not, Is there any other way to watch the file?

 private void FileWatcher()   
     {
         try
         {
             f_Watcher = new System.IO.FileSystemWatcher();
             f_Watcher.Path = InFolder; 
             f_Watcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName;
             f_Watcher.Filter = "*.*";
             f_Watcher.IncludeSubdirectories = true;
             f_Watcher.Created += new FileSystemEventHandler(OnChanged); 
             f_Watcher.EnableRaisingEvents = true;
             f_Watcher.InternalBufferSize = 65536;
             f_Watcher.Error += new ErrorEventHandler(LogBufferError);

         }
         catch (Exception)
         { throw; }
     }

OnChanged Method code block:

private void OnChanged(object sender, FileSystemEventArgs e)
{
    try
    {
        System.Threading.Thread.Sleep(2000);
        InFolder = ConfigurationManager.AppSettings["InputPath"].ToString();
        DirectoryInfo getInDirectoryFile = new DirectoryInfo(InFolder);
        if (!getInDirectoryFile.Exists)
        {
            getInDirectoryFile.Create();
        }

        fileExt = ConfigurationManager.AppSettings["fileExt"].ToString();
        //string file_path = getInDirectoryFile.ToString();
        string file_path = Path.GetDirectoryName(e.FullPath);

        string fname = e.Name.ToString();
        string[] file = Directory.GetFiles(file_path, fileExt); // GET ONLY .txt File
        if (file.Length > 0)
        {

            HLFile_Process.ProcessHL7File(file_path, fname); //Process HL7  files 
        }
        else
        {
            MIEventLogs.WriteLog("Currently there is no file at " + getInDirectoryFile + DateTime.Now.Month.ToString() +
                                 "-" + DateTime.Now.Day.ToString() + "-" + DateTime.Now.Year.ToString() + "-" +
                                 DateTime.Now.Hour.ToString() + ":" + DateTime.Now.Minute.ToString() + ":" +
                                 DateTime.Now.Second.ToString());
        }
    }
    catch (Exception ex)
    {
        MIEventLogs.WriteLog(ex.ToString() + DateTime.Now.Month.ToString() + "-" + DateTime.Now.Day.ToString() + "-" +
                             DateTime.Now.Year.ToString() + "-" + DateTime.Now.Hour.ToString() + ":" +
                             DateTime.Now.Minute.ToString() + ":" + DateTime.Now.Second.ToString());

    }
    finally
    {
        MIEventLogs.WriteLog("MI File Process END: " + DateTime.Now.Month.ToString() + "_" +
                             DateTime.Now.Day.ToString() + "_" + DateTime.Now.Year.ToString() + "_" +
                             DateTime.Now.Hour.ToString() + ":" + DateTime.Now.Minute.ToString() + ":" +
                             DateTime.Now.Second.ToString());
    }
}

Please post all relevant code. This is just the setup for the watcher. That is interesting but rather secondary to the problem. — Fildor
– Fildor, Commented Jul 12, 2023 at 12:36
A generic advice is: Increasing the internal buffer is not going to solve the problem (obviously). You need to make the event handler finish executing as quickly as possible. — Fildor
– Fildor, Commented Jul 12, 2023 at 12:38
What is OnChanged doing? As alternative you can maintain your internal queue (and post there in OnChanged) and process it in parallel. — Guru Stron
– Guru Stron, Commented Jul 12, 2023 at 12:39
@Fildor-standswithMods I have posted all relevant code , Kindly take a look and suggest solution.. — Ravi Kumar
– Ravi Kumar, Commented Jul 13, 2023 at 9:06

Guru Stron · Accepted Answer · 2023-07-13 09:26:51Z

3

First of all remove the System.Threading.Thread.Sleep(2000);. If you have a lot of events this only will lead to the problem you are facing - the internal buffer overflow. If you want to have some "grace" period (i.e. allow to upload more files) - introduce your custom queue and move the "wait" and processing to the dedicated worker(s). For example you can use the ConcurrentQueue<T>, write to it in the OnChanged and run few consumer Tasks which will consume the ConcurrentQueue, perform appropriate waits (await Task.Delay ...) and perform the work ('HLFile_Process.ProcessHL7File').

Also you can consider looking into Rx.NET which can be appropriate for such handling, can throttle, batch and aggregate events.

answered Jul 13, 2023 at 9:26

Guru Stron

151k11 gold badges186 silver badges232 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Fildor Over a year ago

@Ravi: ^^ this, PLUS: take a look into logging in general and "structured logging" in particular. It is not really directly related to the problem you are having, but the code snippet shows a lot of potential on that field.

Fildor Over a year ago

"If you want to have some "grace" period" - this may also cross your way as "debouncing".

Collectives™ on Stack Overflow

FileSystemWatcher InternalBufferSize limit is getting exceeded

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related