Try this:
import re
str_old = str = '$x start $x bar$xy $$x end $x' # You can remove this semicolon
# str = re.sub( '(\s|^)(\$x)(\s|$)' , '$Q', str)
# ^^ You're not placing the spaces back.
str = re.sub( '(\s|^)\$x(\s|$)' , '\\1$Q\\2', str)
print(str)
print(str_old)
That said, you should raw your regex string and replacement string:
str = re.sub(r'(\s|^)\$x(\s|$)' , r'\1$Q\2', str)
And last, avoid using the variable name str in python. There is a function named str already:
import re
str_old = s = '$x start $x bar$xy $$x end $x'
str = re.sub(r'(\s|^)\$x(\s|$)' , r'\1$Q\2', str)
print(str)
print(str_old)
If you don't want to use the backrefereces, you can use lookarounds:
import re
str_old = s = '$x start $x bar$xy $$x end $x' # You can remove this semicolon
str = re.sub(r'(?:(?<=\s)|(?<=^))\$x(?=\s|$)' , '$Q', str)
# Since you don't have backreferences, you can now drop the rawing.
print(str)
print(str_old)
(?:(?<=\s)|(?<=^)) makes sure that the $x is preceded by a space or the beginning of the string;
(?=\s|$) makes sure that the $x is followed by a space or the end of the string.