1

Im trying to create a windows service application with .NET 8 using as a Hosted Service, the idea of my app is Watch for a list of folders. The problem is that I'm losing the event Created fired by the FileSystemWatcher. In context my app should works in this way:

  1. A file is created inside watcher folders.
  2. The file is validated by extension (.txt or .zip) and not empty content.
  3. The file is uploaded to AWS Bucket.
  4. The file is removed from the folder.

These is my happy path, when Im debugging works fine for some files, but if the upload of file is very heavy, then the app lose the events for the FileSystemWatcher.

I'm trying resolve this with Producer/Consumer pattern. I follow these articles:

https://medium.com/@dayanandthombare/background-services-in-net-core-c32f15e28678#:~:text=%E2%9C%85%20Solution%3A%20Background%20Service%20for%20Asynchronous%20File%20Processing

FileSystemWatcher losing files in its queue

But not working to me.

My code looks:

// Worker.cs
public class Worker : BackgroundService
{
private readonly ConcurrentQueue<FileTask> _fileTasks = new ConcurrentQueue<FileTask>();
private readonly SemaphoreSlim _signal = new SemaphoreSlim(0);

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        try
        {
            _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
            _logger.LogInformation("Starting sync files.");

            _synchronizer.SetUp();

            while (!stoppingToken.IsCancellationRequested)
            {
                await _signal.WaitAsync(stoppingToken);

                if (_fileTasks.TryDequeue(out FileTask? task) && task is not null && task.Process is not null && task.FileRelation is not null)
                {
                    await task.Process(task.FileRelation, stoppingToken);
                }

                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
            }
        }
        catch (Exception exception)
        {
            _logger.LogError($"An unhandled error has occurred. Exception {exception}");
            throw;
        }
    }
}

// FileServiceSynchronizer.cs
public class FileServiceSynchronizer : ISynchronizer
{
public void SetUp()
{
 Task.Factory.StartNew(() => 
                    {
                        _logger.LogInformation($"[Watching] {location.FileSystemLocation}");

                        CustomFileSystemWatcher watcher = new CustomFileSystemWatcher(location.FileSystemLocation)
                        {
                            IncludeSubdirectories = true,
                            EnableRaisingEvents = true,
                            Filter = "",
                            CloudLocation = location.CloudLocation,
                            FileSystemErrorLocation = location.FileSystemErrorLocation,
                            NotifyFilter = NotifyFilters.DirectoryName
                                        | NotifyFilters.FileName
                                        | NotifyFilters.LastAccess
                                        | NotifyFilters.LastWrite
                        };

                        watcher.Created += OnFileCreated;
                    });
}

//_fileRead.Process: Validate and Upload to Bucket.
private void OnFileCreated(object sender, FileSystemEventArgs e)
{
worker.EnqueuedFileTask(new FileTask()
                {
                    FileRelation = new RelationLocation()
                    {
                        FileSystemLocation = e.FullPath,
                        CloudLocation = request.CloudLocation,
                        FileSystemErrorLocation = request.FileSystemErrorLocation
                    },
                    Process = _fileRead.Process
                });

}
}

Actually, I'm trying resolve this using Sqlite, where i'm send the event(file path) to database (instead of enqueue file task) and then in the Worker class try to get the paths and process. But I still losing events until nothing get any more.

UPDATE

To works all correctly finally just change my FileSystemWatcher to FileWatcherEx and preserve the pattern used here This Nuget can manage in a best way all events ocurred.

4
  • "The problem is that I'm losing the event Created fired by the FileSystemWatcher" - how do you validate that? Do you have a log event showing that the event was fired but file is not uploaded? Commented Sep 7, 2024 at 4:49
  • Also please add the full code, currently there are a lot of questions. What is worker, what is EnqueuedFileTask, how FileServiceSynchronizer is created and managed? Commented Sep 7, 2024 at 4:51
  • @GuruStron Thanks for you response, now update the post with the solution to my issue. Commented Sep 7, 2024 at 4:54
  • As a side note, the producer-consumer pattern is better implemented with a BlockingCollection<T> or a Channel<T> instead of the ConcurrentQueue<T>. Also instead of the Task.Factory.StartNew it's generally preferable to use the Task.Run. Also it's generally inadvisable to fire-and-forget tasks (as you do with the Task.Factory.StartNew), because any exception will not be observed. Commented Sep 7, 2024 at 6:05

1 Answer 1

1

At least one problem I see is here:

Task.Factory.StartNew(() => 
{
    _logger.LogInformation($"[Watching] {location.FileSystemLocation}");

    CustomFileSystemWatcher watcher = new CustomFileSystemWatcher(location.FileSystemLocation)
    {
        //...
    };

    watcher.Created += OnFileCreated;
});

You are declaring a local variable which will be collected by GC on the next run effectively stopping the filewatcher, assign it to the FileServiceSynchronizer field (and make sure that FileServiceSynchronizer instance is not GCed too):

public class FileServiceSynchronizer : ISynchronizer
{
    private CustomFileSystemWatcher _watcher;

    public void SetUp()
    {
        _watcher = new CustomFileSystemWatcher(location.FileSystemLocation)
        // ...
    }
}

Also there is no reason to use Task.Factory.StartNew here.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.