0

I am creating a bash script that will set a condition to execute a certain command if the line count exceeded per file.

These are the name of the files

dir=/tmp

$dir/filename_$date_00.csv
$dir/filename_$date_01.csv
$dir/filename_$date_02.csv
$dir/filename_$date_03.csv
$dir/filename_$date_04.csv
$dir/filename_$date_05.csv
$dir/filename_$date_06.csv

I want to be able to create a loop with a concept of something like this (i know that the syntax is incorrect but kindly just consider the concept)

if ["cat $dir/filename_$date_00.csv | wc -l" -gt 10]
then 
echo "sorry, cannot proceed"
else,
cat $dir/filename_$date_00.csv >> log..txt
fi

then loop until it reaches $dir/filename_$date_03.csv and ignore $dir/filename_$date_04.csv onwards

I want to be able to create a loop to simplify code instead of printing/repeating everything and have a long code. Please help. thank you!

Improve this question
5
  • 1
    Your question is a little confusing and needs clarification. You seem to have the general idea of how to create the loop (although your syntax is not quite right). Why do you want to ignore files after filename_$date_03.csv? Do you always only process the first 4 files, or does this vary? Commented Mar 16, 2023 at 5:27
  • 1
    Reading and counting all lines of a large file just to check there are more than 10 of them could be wildly inefficient. You could head -n 11 file | wc -l and still check whether it got -gt 10: any further lines are irrelevant. Commented Mar 16, 2023 at 6:28
  • Hi Paul and Merm, I have updated the inquiry. I just want a simplified bash script that has a condition that if filename00 has greater than 10lines, then it will not process the next command, else, it will proceed. Then it will only be applied to the first 4 filenames and will stop with filename03, regardless if there will be filename04.....100. Commented Mar 16, 2023 at 7:08
  • Is the dollar sign part of the filename? If yes, don't forget to escape it, for example with a backslash before the dollar. If the dollar sign indicates a variable, for example date, you have to put curly braces around it ${date}. Otherwise, the underscore and the number are part of the variable name. Commented Mar 16, 2023 at 11:30
  • If you only want to process files 0 to 3, a loop like for i in 0 1 2 3; do # process $dir/filename_${date}_0$i.csv; done will do what you want. Commented Mar 16, 2023 at 11:31

2 Answers 2

Reset to default
1

Here's how I would do it. First, I would let all variables be passed as command line arguments: the name of the target directory, the date, the minimum line number and the maximum number of files. Then, I would use a variable that is incremented by one for every file processed (it isn't clear if files that do not have the right number of lines should be counted, so I am counting them). Something like this:

#!/bin/bash

dir=$1
date=$2
max_files=$3
max_lines=$4

files_seen=0

for file in "$dir"/filename_"${date}"_*.csv; do
  if [[ "$files_seen" -lt "$max_files" ]]; then
    if [[ $(wc -l < "$file") -gt "$max_lines" ]]; then
      cat -- "$file" >> log.txt
    else
      echo "Sorry, cannot proceed with $file"
    fi
    (( ++files_seen ))
  fi
done

To run this on the files in the directory somedir, limiting to only 4 files matching the date 2023-03-16, and with a minimum line number of 10, you would run the script like this:

script.sh somedir 2023-03-16 4 10

If you really do need to limit by the number in the name, then you can use:


#!/bin/bash

dir=$1
date=$2
max_files=$3
max_lines=$4

files_seen=0

for (( i=0; i<$max_files; i++ )); do
  ## zero-pad the number
  num=$(printf '%02d' "$i")
  file="$dir/filename_${date}_${num}".csv; 
  if [[ $(wc -l < "$file") -gt "$max_lines" ]]; then
    cat -- "$file" >> log.txt
  else
    echo "Sorry, cannot proceed with $file"
  fi
done
Improve this answer
0

Here's my take on a solution

#!/bin/sh
dir=/tmp                # Location
date=$(date +%Y%m%d)    # 20230316 for 16 March 2023

# Loop across the first four file numbers
for n in 0 1 2 3
do
    # Generate the matching file name
    file=$(printf "%s/filename_%s_%02d.csv" "${dir:-.}" "$date" $n)

    # Ensure the file exists and it has fewer than 10 lines
    if [ ! -f "$file" ] || [ $(wc -l <"$file") -gt 10 ]
    then
        echo "Sorry, cannot proceed ($file)" >&2
    else
        cat "$file" >>log.txt
    fi
done
Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.