0

I have a script I run regularly using cron. I would like to get notified by email when these scripts fail. I do not wish to be notified every time they run and produce any output at all.

As such, I am using the script Cronic to run my jobs inside cron, which should mean only error output gets sent, and not just any output.

However, in one script I have a command like this:

if [ "$(ls -A ${local_backup_location}/nextcloud-data/)" ]; then
  # save space by removing diffs older than 6 months
  rdiff-backup --remove-older-than 6M --force ${local_backup_location}/nextcloud-data/ || echo "[$(date "+%Y-%m-%d %T")] No existing nextcloud data backup"
fi

The ls -A ${local_backup_location}/nextcloud-data/ is intended to test if a directory is empty. My problem is that this command seems to result in output which is recognized as error output cronic. Cronic defines an error as any non-trace error output or a non-zero result code. For example:

Cronic detected failure or error output for the command:
/usr/local/sbin/run_backup

RESULT CODE: 0

ERROR OUTPUT:
appdata_ocgcv9nemegb
files_external
flow.log
flow.log.1
__groupfolders
.htaccess
index.html
nextcloudadmin
nextcloud-db.bak
nextcloud.log
nextcloud.log.1
.ocdata
rdiff-backup-data
Test_User
updater.log
updater-ocgcv9nemegb ]
custom
gitea-db.sql
log ]
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   365    0     0  100   365      0    302  0:00:01  0:00:01 --:--:--   303
100   365    0     0  100   365      0    165  0:00:02  0:00:02 --:--:--   165
100   365    0     0  100   365      0    113  0:00:03  0:00:03 --:--:--   113
100   365    0     0  100   365      0     86  0:00:04  0:00:04 --:--:--    86
100   365    0     0  100   365      0     70  0:00:05  0:00:05 --:--:--    70
100   365    0     0  100   365      0     58  0:00:06  0:00:06 --:--:--     0
100   365    0     0  100   365      0     50  0:00:07  0:00:07 --:--:--     0
100   365    0     0  100   365      0     44  0:00:08  0:00:08 --:--:--     0
100   365    0     0  100   365      0     39  0:00:09  0:00:09 --:--:--     0
100   365    0     0  100   365      0     37  0:00:09  0:00:09 --:--:--     0
100 10.4M    0 10.4M  100   365  1016k     34  0:00:10  0:00:10 --:--:-- 2493k
100 11.6M    0 11.6M  100   365  1128k     34  0:
 00:10  0:00:10 --:--:-- 3547k

STANDARD OUTPUT:
Maintenance mode enabled
Deleting increment at time:
<snip>

So why does the command ls -A ${local_backup_location}/nextcloud-data/ produce error output in this case, and how can I prevent this? An alternative robust method to test if a directory is empty would be acceptable, but I would also like an explanation of why the command seems to produce error output.

EDIT: Adding Cronic stdout with set -ex

Some commenters have requested the actual whole script which is very long, but Cronic reports the actual stdout of the script and I use set -ex at the top of the script. The error output happens immediately after the invocation of ls -A /mnt/reos-storage-2/backups/nextcloud-data/ which is why I believe the error output to be the result of this command.

+ rdiff-backup --ssh-no-compression /var/www/nextcloud /mnt/reos-storage-2/backups/nextcloud/
+ ls -A /mnt/reos-storage-2/backups/nextcloud-data/
+ [ 67cf481e-62a3-1039-8bf2-05805d214bca
<removed>
appdata_ocgcv9nemegb
<removed>
<removed>
<removed>
<removed>
files_external
flow.log
flow.log.1
__groupfolders
.htaccess
index.html
<removed>
<removed>
nextcloudadmin
nextcloud-db.bak
nextcloud.log
nextcloud.log.1
.ocdata
<removed>
<removed>
rdiff-backup-data
<removed>
Test_User
<removed>
updater.log
updater-ocgcv9nemegb ]
+ rdiff-backup --remove-older-than 6M --force /mnt/reos-storage-2/backups/nextcloud-data/
+ date +%Y-%m-%d %T
+ echo [2021-04-21 03:23:38] Starting nextcloud data backup
Improve this question
14
  • 1
    You say you have a command like that. The command that you show does not produce output. Can you please double check that the actual command is identical to what you show in the question? It looks as if the error output contains an unsorted list of files, which indicates that it's not a list produced by ls. Are you in fact calling find? Commented Apr 23, 2021 at 15:55
  • @Kusalananda it's plausible output for some locales that don't sort non-alphanumerics Commented Apr 23, 2021 at 17:20
  • @roaima Well, it's not the shown command that produces the specific output in any case. Notice how the command substitution is quoted and how the error message contains ] twice and no actual error message. There's also output from curl (?), but no curl command is shown. Commented Apr 23, 2021 at 17:28
  • Ah yes. The last three filenames (custom, gitea-db.sql, log) are out of sequence but the order of the remainder is reproducible under (at least) en_GB.UTF-8. Commented Apr 23, 2021 at 17:39
  • 1
    @crobar, remove any irrelevant parts from that script until you're left with just the part that causes the error. Then post that remaining script in full, including the #!/bin/bash line and any set -o errexit lines etc, and also the cron command you use to launch it. If you think the problem is in the part you quoted, then leave just that, and post the results: the full script and the error output you get. And if the problem doesn't appear after you remove everything else, well, then the problem is somewhere else. Commented Apr 23, 2021 at 19:25

2 Answers 2

Reset to default
1 
+ [ 67cf481e-62a3-1039-8bf2-05805d214bca
<removed>
[...]
updater.log
updater-ocgcv9nemegb ]

This here, is one single command. It's split to multiple lines in the set -x/xtrace output, because the output from ls contains newlines. (Bash would print that with some quotes, Dash doesn't.)

By default, the xtrace output goes to stderr, the same as the regular error output, Cronic tries to separate them by looking at the + marker at the start of the xtrace lines. That fails here, and it thinks the files names there are part of regular error output.

What cronic does is basically this:

PATTERN="^${PS4:0:1}\\+${PS4:1}"
if grep -aq "$PATTERN" $TRACE
then
    ! grep -av "$PATTERN" $TRACE > $ERR

Disabling xtrace would be one way to fix that, but it would be a shame to do that since cronic supports it so nicely.

Instead, it might be better to use some other way to check if the directory is empty.

Keeping with ls -A, you could pipe the output to wc to count the characters there:

if [ "$(ls -A "${local_backup_location}/nextcloud-data/" | wc -c)" -gt 0 ]; then
    echo "directory not empty"
fi

or to grep:

if ls -A "${local_backup_location}/nextcloud-data/" | grep -q .; then
    echo "directory not empty";
fi

Checking if the directory is empty could be done in other ways, within the shell itself, but handling all the corner cases can be hairy. See e.g. Portable check empty directory

Improve this answer
2
  • nullglob and dotglob needed for that last bit in bash. Commented Apr 26, 2021 at 17:06
  • 1
    @Kusalananda, bahh, yes, I wasn't focusing. And they don't even seem to be using Bash, so perhaps ls -A will have to do. Commented Apr 26, 2021 at 19:37
-1

If $gitea_backup_dir doesn't exist, or makes ls unhappy in any way, ls will write an error message to STDERR.

You can discard the whole error stream by appending 2>/dev/null to your command.

IMHO, a better way is:

if ([[ -d "$gitea_backup_dir" ]] && \
    [[ $(stat --format="%h" "$gitea_backup_dir") -gt 2 ]] ) ; then
    echo "Nonempty"
else
    echo "Empty"
fi

First, check that the directory exists, then see if the number of hard links in the directory (the number of file and subdirectory entries in the directory) is greater than 2 (every directory has at least 2 entries: . and ..).

Improve this answer
4
  • Or set -- "$gitea_backup_dir"/*; if [ "$#" -gt 0 ]; then ...; fi (assuming nullglob and dotglob are set). Commented Apr 23, 2021 at 16:36
  • looking at the link count can tell if there's other directories inside the directory, but not files, while the ls -A would produce nonempty output if there's anything in the directory, regular files, directories or otherwise, right? Commented Apr 23, 2021 at 19:12
  • Both files in a directory and (sub-)directories in a directory ARE hard links. Don't ask me what ls -A would do. It's easy to set up a test environment and try it. Or read man ls. Commented Apr 23, 2021 at 19:17
  • @waltinator, (was that a reply to me? Use the @-notation, so that people get notifications.) They said that the ls -A "is intended to test if a directory is empty", and that's what it does in that if there is at least file in the directory, it prints the name. If there isn't, the output is empty. And yes, files are hard links, but they're hard links to themselves, not to the directory. Try e.g. mkdir foo; ls -ld foo (the link count should be 2), touch foo/bar; ls -ld foo (the link count is still 2), and mkdir foo/doo; ls -ld foo (now the link count is 3) Commented Apr 26, 2021 at 13:19

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.