The difference is that you constrain the two character to be exactly the same.
In the first regular expression there must not be any non-whitespace character following:
/\S(?![\s\S]*\S)/
This basically means to match any non-whitespace character that is not followed by any other non-whitespace character. That’s why it matches the last o that is only followed by whitespace:
"Hello World\nHello "
^ no other non-whitespace character following
But in the second regular expression there must not be that specific character following that was matched before:
/(\S)(?![\s\S]*\1)/
This basically means to match any non-whitespace character that does not appear again in the rest of the string. That’s why the W is matched as it’s the first non-whitespace character that does not appear again after the first occurrence:
"Hello World\nHello "
^ no other “W” following
That’s why it’s called a backreference: You reference a previously matched string.