Error with a script in bash

Question

I have a little error with a script I wrote in bash and I can't figure out what's I'm doing wrong

note that I'm using this script for thousands of calculations and this error happened only a few times (like 20 or so), but it still happened

What the script does is this: basically it takes in input a web page that I got from a site with the utility w3m and it counts all the occurrences of the words in it... After it orders them from the most common to the ones that occur only once

this is the code:

#!/bin/bash
#   counts the numbers of words from specific sites                       #
#   writes in a file the occurrences ordered from the most common         #

touch check         # file used to analyze the occurrences
touch distribution      # final file ordered

page=$1             # the web page that needs to be analyzed
occurrences=$2          # temporary file for the occurrences
dictionary=$3                       # dictionary used for another purpose (ignore this)

# write the words one by column
cat $page | tr -c [:alnum:] "\n" | sed '/^$/d' > check

# lopp to analyze the words
cat check | while read words
do
    word=${words}
    strlen=${#word}
    # ignores blacklisted words or small ones
    if ! grep -Fxq $word .blacklist && [ $strlen -gt 2 ]
    then
        # if the word isn't in the file
        if [ `egrep -c -i "^$word: " $occurrences` -eq 0 ]
        then
            echo "$word: 1" | cat >> $occurrences
        # else if it is already in the file, it calculates the occurrences
        else
            old=`awk -v words=$word -F": " '$1==words { print $2 }' $occurrences`
                    ### HERE IS THE ERROR, EITHER THE LET OR THE SED ###
            let "new=old+1"
            sed -i "s/^$word: $old$/$word: $new/g" $occurrences
        fi
    fi
done

# orders the words
awk -F": " '{print $2" "$1}' $occurrences | sort -rn | awk -F" " '{print $2": "$1}' > distribution

# ignore this, not important
grep -w "1" distribution | awk -F ":" '{print $1}' > temp_dictionary

for line in `cat temp_dictionary`
do
    if ! grep -Fxq $line $dictionary
    then
        echo $line >> $dictionary
    fi
done

rm check
rm temp_dictionary

this is the error: (I'm translating it, so it could be different in english)

./wordOccurrences line:30 let:x // where x is a number, usually 9 or 10 (but also 11, 13, etc)
1: syntax error in the espression (the error token is 1)
sed: expression -e #1, character y: command 's' not terminated // where y is another number (this one is also usually 9 or 10) with y being different from x

EDIT: Talking with kev it looks like it's a newline problem

I added an echo between let and sed to print the sed and it worked perfectly for like 5 to 10 minutes until that error. Usually the sed without error looked like this:

s/^CONSULENTI: 6$/CONSULENTI: 7/g

but when I got the error it was like this:

s/^00145: 1 1$/00145: 4/g

how to fix this?

Nahuel Fouilleul · Accepted Answer · 2012-08-05 13:35:50Z

2

If you get a new line in $old, it means awk prints two lines so there is a duplicate in $occurences.

The script seems complicated to count words, and not efficient because it launches many processes and process file in a loop ; maybe you can do something similar with

sort | uniq -c

answered Aug 5, 2012 at 13:35

Nahuel Fouilleul

19.5k2 gold badges36 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

cds · Accepted Answer · 2012-08-05 14:19:34Z

You should also consider that your case-insensitivity is not consistent throughout the program. I created a page with just "foooo" in it and ran the program, then created one with "Foooo" in it and ran the program again. The 'old=`awk...' line sets 'old' to the empty string because awk is matching case sensitively. This results in the occurrences file not being updated. The subsequent sed and possibly some of the greps are also case sensitive.

This may not be the only error since it doesn't explain the error message you saw, but it is an indication that the same word with different capitalization will be handled erroneously by your script.

The following would separate the words, lowercase them, and then remove the ones smaller than three characters:

tr -cs '[:alnum:]' '\n' <foo | tr '[:upper:]' '[:lower:]' | egrep -v '^.{0,2}$'

Using this at the front of your script would mean that the rest of the script would not have to be case insensitive to be correct.

Collectives™ on Stack Overflow

Error with a script in bash

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related