1

Is there a memory efficient way to use 'using' within a recursive function when e.g. writing lines to a file?

I read C# 'using' block inside a loop and it mentioned that you don't want to put a using statement inside a for loop unless you have to. (makes sense, one doesn't want multiple instances of 'using' if one doesn't need them). So in the case of a for loop if you can put it outside, you do.

But here, I have a recursive function. So the 'using' statement is going to run multiple times even if I put it outside of a for.

So what is a good or proper way of placing the 'using' statement?

I don't know if I should avoid 'using', and declare the StreamWriter object, StreamWriter writetext before the method call and dispose of it after with writetext.Dispose(). Or maybe there is a more conventional way with 'using'. Maybe wrapping the 'main' call DirSearch_basic_writetofile("c:\\aaa"); with a 'try' and putting the Dispose line in a finally. And avoiding 'using' then. That's just a thought.

// requires directory c:\texts
File.Delete(@"c:\texts\filelist.txt");
// list files and subdirectories of c:\aaa and write them to file "c:\texts\filelist.txt"
DirSearch_basic_writetofile("c:\\aaa");

// recursive function that lists files  and directories and subdirectories,in given directory  

static void DirSearch_basic_writetofile(string sDir)
{
    Console.WriteLine("DirSearch(" + sDir + ")");
    Console.WriteLine(sDir+@"\");
    try
    {
        using (StreamWriter writetext = new StreamWriter("c:\\texts\\filelist.txt",true))
        {
            writetext.WriteLine("DirSearch(" + sDir + ")");
            writetext.WriteLine(sDir);

            foreach (string f in Directory.GetFiles(sDir))
            {
                Console.WriteLine(f);
                writetext.WriteLine(f);
            }
        }

        foreach (string d in Directory.GetDirectories(sDir))
        {
            DirSearch_basic_writetofile(d);
        }

    }
    catch (System.Exception excpt)
    {
        Console.WriteLine(excpt.Message);
    }

}
Improve this question
10
  • Pass the streamwriter down the recursive calls. Commented May 15, 2020 at 2:15
  • Also, consider a different approach. Have one thread populating a BlockingCollection with entries of file / directory names. Then have a second thread reading from the BlockingCollection (basically a Producer Consumer model). That way you don't even need to pass the StreamWriter around (since it is used in only one place). Commented May 15, 2020 at 2:17
  • @mjwills re "Pass the streamwriter down the recursive calls. " <-- I suppose that'd be one way of a general solution of declaring the Stream outside of the recursive function. e.g. using(StreamWriter writetext = new StreamWriter("c:\\texts\\filelist.txt", true)) DirSearch_basic_writetofile("c:\\aaa"); (not passing the stream down the recursive calls), but still , declaring it outside of it. Commented May 15, 2020 at 2:17
  • 1
    Note the bottleneck in your project is likely to be the Console.WriteLine calls. If you wanted to improve performance you should reduce the number of Console.WriteLine calls (e.g. call it every 10 or 100 or 1000 entries (e.g. concatenating 100 entries and then writing them all to the console in one hit) rather than every entry). Commented May 15, 2020 at 2:21
  • 1
    "Because it is" (mainly due to locking). Try it and see. stackoverflow.com/a/50979627/34092 Commented May 15, 2020 at 2:37

3 Answers 3

Reset to default
1

The linked thing is a case where you were using the same resource for all itterations of the loop. In that case, opening and closing it every itteration serves no purpose. As long as it is closed by the end of all loops, it is save enough.

The opposite case is if you use a different resource every itteration. Say, when going over a list of filenames or full paths, to open each in turn. In that case you got no chocie but to have a new File related instance each itteration.

A recursion is not really different from a loop. You can always replace a loop with a recursion, but the opposite is not always the case. Same rules apply:

  • If it is the same resource, you just have to move the creation of the resource outside the recursive function. Rather then taking a path (or using a hardcoded one), let it take a Stream. That keeps the function nicely generic
  • If you got a different resource, you have no choice but create a new Instance with a new using every recursion. However I can not think of any "recursive using" case.

If you got to itterate over all files in a directory inlcuding all subdirectories, you would have the recursive function recurse over the directories (no unmanaged resource needed). And then a loop inside the recursive function to itterate over the files inside the current directory (wich requires unmanaged resources).

Edit:

static void DirSearch_basic_writetofile(string currentDirectory, StreamWriter Output){
    //do your thing, using output as the stream you write to

    //Do recusirve calls as normal
    DirSearch_basic_writetofile(subDir, Output);
}

calling it:

using (StreamWriter OutputWriter = new StreamWriter("c:\\texts\\filelist.txt",true){
    DirSearch_basic_writetofile(startDirectory, OutputWriter);
}
Improve this answer
Sign up to request clarification or add additional context in comments.

22 Comments

Actually any recursive code can be rewritten with loops and a Stack.
@juharr Recursion is just calling the same code over and over, using a call stack. So it sounds like you reinvented the call stack.
More or less, but that's still a way to translate recursive code into looping code.
Converting a recursive function to a state machine can expose more opportunities for compiler optimisations. When you get down to the bare metal of assembly, all control flow is just GOTO's with extra steps.
@barlop It can be better to you only hand a String Builder through (instead of teh string), defering the writing of the file until you finished all loops. But taht would make this "all or nothing".
|
1

If we want to solve using yield return

You might want to restructure the code such that you separate out the recursive part; for example with a yield return.

Something like this below ( sorry, no IDE at hand, let's see if this works) is a simplistic approach.

If you need to write out the new header ( DirSearch(" + sDir + ") ) every time you switch directory, that's doable by not returning String only from producer an object containing String directoryName, List fileNames, and return only once for each directory.

static void DirSearch_basic_writetofile(string sDir)
{
    Console.WriteLine("DirSearch(" + sDir + ")");
    Console.WriteLine(sDir+@"\");
    IEnumerable<String> producer = DirSearch_Producer(string sDir);
    try
    {
        using (StreamWriter writetext = new StreamWriter("c:\\texts\\filelist.txt",true))
        {
            writetext.WriteLine("DirSearch(" + sDir + ")");
            writetext.WriteLine(sDir);

            foreach (string f in DirSearch_Producer(sDir))
            {
                Console.WriteLine(f);
            }
        }
    }
    catch (System.Exception excpt)
    {
        Console.WriteLine(excpt.Message);
    }
}

public static IEnumerable<String> DirSearch_Producer(string sDir){
  foreach (string f in Directory.GetFiles(sDir))
  {
    yield return f;
  }
  foreach (string d in Directory.GetDirectories(sDir))
  {
    foreach (String f in DirSearch_Producer(d)){
        yield return f;
    }
  }
}

Alternative, without using yield return we can use the Directory.GetFiles with EnumerationOptions to go through subdirectories as well. It makes things much simpler. See: RecurseSubdirectories

Improve this answer

4 Comments

Is there perhaps an alternative to GetFiles and GetDirectories that already does this (i.e. avoids the need for yield return)?
Right, actually there is :) We can use the Directory.GetFiles with EnumerationOptions to go through subdirectories as well. It makes things much simpler. See: RecurseSubdirectories
@TomS. I think you should add this to your answer as an alternative.
Also check EnumerateFiles rather than GetFiles. learn.microsoft.com/en-us/dotnet/api/…
0

I am going to leave Christopher's answer as accepted..

I will just note here the solution I used..

using (writetext = new StreamWriter("c:\\texts\\filelist.txt", true))
   DirSearch_basic_writetofile_quicker("c:\\aaa");

I also used a StringBuilder and wrote to console and file after

        // https://stackoverflow.com/questions/15443739/what-is-the-simplest-way-to-write-the-contents-of-a-stringbuilder-to-a-text-file
        System.IO.File.WriteAllText(@"c:\texts\filelist.txt", sb.ToString());

        Console.Write(sb.ToString());

mjwillis had suggested to me that writing to the console one line at a time was a bottleneck, and christopher mentioned about writing to the file at the end. So I just write to both at the end / after the call.

I used a static variable for StreamWriter and a static variable for StringBuilder, so that main() and the recursive function could see it. I didn't want to create a new parameter for the recursive function, 'cos I wanted to keep the recursive call looking 'simple' (one parameter).

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.