read disables the terminal echo only for the duration of that read command. So if you type something in between two read commands, the terminal driver will echo it back.

You should disable echo and then call read in your loop:

if [ -t 0 ]; then
  saved=$(stty -g)
  stty -echo
fi
while read -rN1; do
  ...
done
if [ -t 0 ]; then
  stty "$saved"
fi

read -s disables the terminal echo only for the duration of that read command. So if you type something in between two read commands, the terminal driver will echo it back.

You should disable echo and then call read in your loop without -s:

if [ -t 0 ]; then
  saved=$(stty -g)
  stty -echo
fi
while read -rN1; do
  ...
done
if [ -t 0 ]; then
  stty "$saved"
fi