out of memory exception while using multithread in C#

Question

I am new to threading and stuck here on this issue. I have an application where I am using multi threading. I have a function which uploads thousands of images on ftp server. For each images I am creating a new thread. This thread calls a function to connect ftp server, upload file and returns boolean value if its successfully uploaded.

My problem is, since I am uploading thousands of images and each is creating its own thread, after sometime I get Out of Memory exception and application freezes.

My codes is as follows:

public Int16 UploadFiles(string[] files)
{
     foreach (var fileName in files)
        {
            if (UploadFile(fileName))
            {
                strLogText += "\r\n\tFile: " + fileName + " downloaded.";
            }
        }
}

private bool UploadFile(string fileName)
    {
        var blnDownload = false;
        var thread = new Thread(() => DownLoadFileNow(fileName, out blnDownload)) {IsBackground = true};
        thread.Start();
        return blnDownload;
    }

    private void DownLoadFileNow(string fileName, out bool blnDownload)
    {
        //Get file path and name on source ftp server
        var srcFolder = GetSrcFolderName(fileName);

        //Get Local Folder Name for downloaded files
        var trgFolder = GetLocalFolder(fileName, "D");

        var reqFtp =
            (FtpWebRequest) WebRequest.Create(new Uri("ftp://" + _strSourceFtpurl + srcFolder + "/" + fileName));
        reqFtp.Method = WebRequestMethods.Ftp.DownloadFile;
        reqFtp.UseBinary = true;
        reqFtp.Credentials = new NetworkCredential(_strSourceFtpUser, _strSourceFtpPassword);
        var outputStream = new FileStream(trgFolder + "\\" + fileName, FileMode.Create);

        try
        {
            var response = (FtpWebResponse) reqFtp.GetResponse();
            var ftpStream = response.GetResponseStream();
            const int bufferSize = 2048;
            var buffer = new byte[bufferSize];

            if (ftpStream != null)
            {
                int readCount = ftpStream.Read(buffer, 0, bufferSize);
                while (readCount > 0)
                {
                    outputStream.Write(buffer, 0, readCount);
                    readCount = ftpStream.Read(buffer, 0, bufferSize);
                }

                ftpStream.Close();
            }
            response.Close();
            blnDownload = true;
        }
        catch (WebException ex)
        {
            _log.WriteLog("Error in Downloading File (" + fileName + "):\r\n\t" + ex.Message, "");
            //Delete newly created file from local system
            outputStream.Close();
            if (File.Exists(trgFolder + "/" + fileName))
                File.Delete(trgFolder + "/" + fileName);
        }
        catch (Exception ex)
        {
            _log.WriteLog("Error in Downloading File (" + fileName + "):\r\n\t" + ex.Message, "");
        }
        finally
        {
            outputStream.Close();
            outputStream.Dispose();
        }
        blnDownload = false;
    }

Please help and let me know how I can limit the number of threads being created so that at a time, there are not more than 10-20 threads running.

a thread uses 1MB of memory; that's why you get the error. You need to use TPL instead, as suggested in the answers below. — Cristian Lupascu
– Cristian Lupascu, Commented Jul 25, 2013 at 15:39
Make a kind of queue, that limits how many things you can download at a time. Maybe try using Tasks instead of threads so you can know when they finish — Robert Hoffmann
– Robert Hoffmann, Commented Jul 25, 2013 at 15:39
Any Windows process that uses more than (8 * logical_processors) of threads will fail to perform or scale. For example, a single CPU system with an service application that uses more than 8 thread to process requests (e.g. socket accepts), will quickly take the box to its knees. — user83106
– user83106, Commented Jul 25, 2013 at 15:48
@DanielBullington - what? I'm posting this from Firefox - 57 threads. My i7 has 4/8 cores. Kaspersky AV has 92 threads. NT kernel and System has 257 threads. Where did you get that 'more than (8 * logical_processors)' fallacy from? — Martin James
– Martin James, Commented Jul 25, 2013 at 16:20
The OP's problem is continually creating/terminating/destroying threads instead of submitting tasks to a threadpool. For networked tasks that are mostly blocked on I/O, 64 threads in the pool is not unreasonable. — Martin James
– Martin James, Commented Jul 25, 2013 at 16:25

Hylaean · Accepted Answer · 2013-07-25 15:37:58Z

3

You can't create this many threads. One alternative would be to use parrelle extensions.

public void UploadFiles(string[] files)
{
    files.AsParallel().ForAll(fileName =>
    {
        if (UploadFile(fileName))
        {
            strLogText += "\r\n\tFile: " + fileName + " downloaded.";
        }
    });
}

answered Jul 25, 2013 at 15:37

Hylaean

1,29713 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user467384 · Accepted Answer · 2013-07-25 15:37:38Z

2

Try replacing the foreach in UploadFiles(string[] files) with a Parallel.ForEach() that calls DownloadFileNow instead of the UploadFile(String file) method.

Parallel.Foreach will pull threads from a thread pool, which is what you want, and will simplify your code.

 Parallel.ForEach(files, fileName =>
        DownloadFileNow(fileName);
        strLogText += "\r\n\tFile: " + fileName + " downloaded.";
 );

answered Jul 25, 2013 at 15:37

user467384

1,1674 gold badges23 silver badges40 bronze badges

7 Comments

sarojanand Over a year ago

Thanks, I am now getting "AggregateException was unhandled". any idea how to resolve this?

user467384 Over a year ago

I bet it has to do with updating the strLogText in an asynchronous manner. What do you do with strLogText?

sarojanand Over a year ago

I use to add in the log file at the end of the function to know how many files and which files are uploaded. instead of adding each time, I add them in string and after each loop ends, I add in log file.

sarojanand Over a year ago

I commented that line for strLogText, still I can see the same error.

user467384 Over a year ago

Did it give you a line number in the AggregateException stack trace?

|

Daren Thomas · Accepted Answer · 2013-07-25 15:47:03Z

As others have pointed out, you shouldn't be creating that many threads. Now... The Parallel.ForEach() will give you nice syntactic sugar and you should go that route!

Just wanted to point out, that you need to look up thread pools as a concept. See, it doesn't make sense to have tons of threads running around in parallel. For each such task, there is an optimum number of threads, above which, the whole threading overhead will actually start to slow you down. Or, in your case, use up all your memory.

If you think about the task as a pile of photos sitting on a desk (the folder) and the threads as employees running errands for you, then having one employee fetch a photo from the desk, put it in an envelope and bring it to the post office, for each photo, is going to take forever. So you hire another employee. And another. But, once you reach a certain amount of photo-stamp-post-guys, they start to get in each others way. They queue up in front of the desk. They queue up in front of the post office. And the whole envelope situation is just painful for poor Mary in charge of office supplies - they're queuing up in front of her too! So, figure out the optimal amount of employees (you could run tests or just guess a bit) and assign them to repeat that task until the desk is empty...

This is a master / slave pattern that shows up quite often.

Collectives™ on Stack Overflow

out of memory exception while using multithread in C#

3 Answers 3

Comments

7 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

7 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related