Looping all .csv files in one directory through a script, and output the results of each loop into a separate .csv in different directory

Question

I am struggling to run multiple .csv files from one directory through a script, and sending the output of each to a different directory. I am able to run the .csv files one by one, but I have 100 files that need to go through the script. Individually, it's pretty simple, I am pulling my .csv file from Test_Folder and sending the output of the script to Results:

bash run_example.sh Test_Folder/df1.csv >> Results/results1.csv

But when I attempt to run a loop for them all, I get met with some errors due to having different directories. I've tried a few different things:

for csvfile in Test_Folder/*.csv;
do
    bash run_example.sh $csvfile >> Results/test_results.csv
done

This above obviously works, but test_results.csv gets overwritten after each loop.

for csvfile in Test_Folder/*.csv;
do
    bash run_example.sh $csvfile >> "Results/results_${csvfile}.csv"
done

When trying to run the above, I get met with:

bash: Results/results_Test_Folder/df1.csv.csv: No such file or directory

So it appears that it's taking the path of $csvfile rather than just the file itself. This ends up searching for a directory that clearly doesn't exist. I have a tried a few other variations of the above code, using other SE posts as reference, but I have not seen any post that specifically mentions different directories.

I'm looking for the new directory to be populated with the same amount of results.csv files as input files, so results1.csv, results2.csv, and so on. Even something as simple as a number that gets incremented with each loop and added to the output file name would be perfect. Any guidance would be appreciated!

bash run_example.sh $csvfile >> Results/${csvfile#Test_Folder/}.csv -- you need to strip "Test_Folder/" from the front of your filename when looping for csvfile in Test_Folder/*.csv; The variable csvfile will hold "Test_Folder/filename.csv" where you want the output filename to be "Result/filename.csv" The parameter-expansion with substring removal will do that for you.. — David C. Rankin
– David C. Rankin, Commented Jul 14, 2021 at 23:13

David C. Rankin · Accepted Answer · 2021-07-15 04:38:04Z

Presuming the comment solved your problem, for completeness, the problem you experienced is due to the loop variable csvfile containing the path information for Test_Folder/. When you redirect to Results/results_${csvfile}.csv, you are actually redirecting to, e.g. Results/results_Test_Folder/df1.csv.csv. Where you haven't trimmed Test_Folder/ from the front of csvfile and you have applied the .csv extension twice. This results in your exact error:

bash: Results/results_Test_Folder/df1.csv.csv: No such file or directory

To correct the problem you can use parameter-expansion with substring removal to trim Test_Folder/ from the front of csvfile and since the filename already has the .csv extension, simply omit it, e.g.

for csvfile in Test_Folder/*.csv;
do
    bash run_example.sh "$csvfile" >> "Results/results_${csvfile#Test_Folder/}"
done

This would result in the redirection for the example you gave for df1.csv to:

Results/results_df1.csv

(note: you should quote the original $csvfile in bash run_example.sh "$csvfile" ...)

The standard parameter-expansions with substring removal provided by POSIX are:

${var#pattern}      Strip shortest match of pattern from front of $var
${var##pattern}     Strip longest match of pattern from front of $var
${var%pattern}      Strip shortest match of pattern from back of $var
${var%%pattern}     Strip longest match of pattern from back of $var

Note, bash itself provides many, many, many more. See man 1 bash under the Parameter Expansion heading.

Look things over and let me know if you have further questions.

Collectives™ on Stack Overflow

Looping all .csv files in one directory through a script, and output the results of each loop into a separate .csv in different directory

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related