BASH: Loops through CSV and cURL rows with grep [duplicate]

Question

I am aiming to loop through a CSV file in bash, i have some code that is semi working. It does not seem to be completing the cURLs as the command instantly finishes and adds to the fail file. It is reading a single column CSV of URLs.

I have researched with no success, looking at GREP success codes etc. This is just being ran locally on MACOS.

INPUT=test.csv
OLDIFS=$IFS
IFS=,
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
while read col1
do
    if curl -L -s "$col1" | grep -q "string"; then
        echo "${col1}" >> one.txt
    else
        echo "${col1}" >> two.txt
    fi
done < $INPUT
IFS=$OLDIFS

I have also tried the following on the IF line: if [[ $(curl -L -s "$col1" | grep -q "string") ]]; then

Thanks

Let's try to debug it. Immediately after the do add a line echo $col1 so that you can see if what you get from reading test.csv is really what you want. — gboffi
– gboffi, Commented Oct 3, 2018 at 14:14
echoing col1 gives me the domains on new lines, i have also added leftover as per @PTRK instruction. — Aaron Nergaard
– Aaron Nergaard, Commented Oct 3, 2018 at 14:26
Please modify your question to add the output of your script with few lines of test.csv. — PTRK
– PTRK, Commented Oct 3, 2018 at 14:28
After removing the -s i get the following error curl: (3) Illegal characters found in URL so it must be the URL in cURL — Aaron Nergaard
– Aaron Nergaard, Commented Oct 3, 2018 at 14:32

PTRK · Accepted Answer · 2018-10-03 14:22:01Z

1

From the bash manual

One line is read from the standard input, or from the file descriptor fd supplied as an argument to the -u option, and the first word is assigned to the first name, the second word to the second name, and so on, with leftover words and their intervening separators assigned to the last name

since you only provide one name, col1 is not the first column, but the whole line.

Try while read col1 leftover and use @gboffi's comment advice

answered Oct 3, 2018 at 14:22

PTRK

8991 gold badge5 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Aaron Nergaard · Accepted Answer · 2018-10-03 14:51:45Z

0

Fixed with the following:

col1=${col1%$'\r'} - to remove the full stops from output.

INPUT=test-sites.csv
OLDIFS=$IFS
IFS=,
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
while read col1 leftover
do
    col1=${col1%$'\r'}
    if curl -L "${col1}" | grep -q "string"; then
        echo "${col1}" >> one.txt
    else
        echo "${col1}" >> two.txt
    fi
done < $INPUT
IFS=$OLDIFS

Thank you for your comments

answered Oct 3, 2018 at 14:51

Aaron Nergaard

267 bronze badges

1 Comment

tripleee Over a year ago

Those are carriage returns, not "full stops". The proper solution is to not use a Windows editor.

Collectives™ on Stack Overflow

BASH: Loops through CSV and cURL rows with grep [duplicate]

2 Answers 2

Comments

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Linked

Related