4

I am making a shell script that runs a bunch of tests as part of a CI pipeline. I would like to run all of the tests (I DO NOT WANT TO EXIT EARLY if one test fails). Then, at the end of the script, I would like to return with a negative exit code if any of the tests failed.

Any help would be appreciated. I feel like this would be a very common use case, but I wasn't able to find a solution with a bit of research. I am pretty sure that I don't want set -e, since this exits early.

My current idea is to create a flag to keep track of any failed tests:

flag=0
pytest -s || flag=1
go test -v ./... || flag=1
exit $flag

This seems strange, and like more work than necessary, but I am new to bash scripts. Am I missing something?

Improve this question
6
  • What you have is a good approach. What is failing in it? Replace $flag with just flag. Commented Jul 6, 2018 at 7:35
  • How are the tests run? in a loop? Commented Jul 6, 2018 at 7:37
  • use ... || flag=1 (no need for $) Commented Jul 6, 2018 at 7:37
  • It seems excessively wordy to have to specify this for every single line in the script. Right now, I'm only running a few commands, but I am afraid that this won't scale well if more tests are added in the future. Commented Jul 6, 2018 at 7:37
  • thanks for the feedback! I made the edits. I am not running the tests in a loop. Commented Jul 6, 2018 at 7:39

3 Answers 3

Reset to default
4

One possible way would be to catch the non-zero exit code via trap with ERR. Assuming your tests don't contain pipelines | and just return the error code straight to the shell launched, you could do

#!/usr/bin/env bash

exitCodeArray=()

onFailure() {
    exitCodeArray+=( "$?" )
}

trap onFailure ERR

# Add all your tests here

addNumbers () {
    local IFS='+'
    printf "%s" "$(( $* ))"
}

Add your tests anywhere after the above snippet. So we keep adding the exit code to the array whenever a test returns a non-zero return code. So for the final assertion we check if the sum of the array elements is 0, because in an ideal case all cases should return that if it is successful. We reset the trap set before 

trap '' ERR

if (( $(addNumbers "${exitCodeArray[@]}") )); then
    printf 'some of your tests failed\n' >&2
    exit -1
fi
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Why do we need the array here? Why not use a simple integer in onFailure that sums up the exit codes?
@codeforester: Good catch! If I remember right, I wanted the array, so that if needed we can print the array to see the status of individual tests from the start and map each test with its exit code. If one doesn't need that, you could use a variable
2

I vote for Inian's answer. Traps seem like the perfect way to go.

That said, you might also streamline things by use of arrays.

#!/usr/bin/env bash

testlist=(
  "pytest -s"
  "go test -v ./..."
)

for this in "${testlist[@]}"; do
  $this || flag=1
done

exit $flag

You could of course fetch the content of the array from another file, if you wanted to make a more generic test harness that could be used by multiple tools. Heck, mapfile could be a good way to populate an array.

Improve this answer

Comments

1

The only way I could imagine using less code is if the shell had some sort of special all compound command that might look something like

# hypothetical all command
all do
  pytest -s
  go test -v ./...
done

whose exit status is the logical or of the exit statuses of the contained command. (An analogous any command would have the logical and of its commands' exit statuses as its own exit status.)

Lacking such a command, you current approach is what I would use. You could adapt @melpomene's suggestion of a chk function (which I would call after a command rather than having it call your command so that it works with arbitrary shell commands):

chk () { flag=$(( flag | $? )); }

flag=0
pytest -s; chk
go test -v ./...; chk
exit "$flag"

If you aren't using it for anything else, you could abuse the DEBUG trap to update flag before each command.

trap 'flag=$((flag | $?))' DEBUG
pytest -s
go test -v ./...
exit "$flag"

(Be aware that a debug trap executes before the shell executes another command, not immediately after a command is executed. It's possible that the only time this matters is if you expect the trap to fire between the last command completing and the shell exiting, but it's still worth being aware of.)

Improve this answer

2 Comments

Wouldn't it make more sense for the hypothetical all command to be the logical AND, and the any command to be the OR? Or did I somehow read that wrong?
I'm using all to mean "all must succeed", not "all must fail". For example, four exit statuses of 0, 0, 1, and 0 should produce a 1 (the block fails if any command fails).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.