If you really want to do it in one step you'll have to do multiple lookaheads/lookbehinds to account for all cases (and it's a question if all of them are even captured with this one):
import re
my_str = '\'1.5"x3"x10" hey 7" x 4"x 2" how 9.5" x 9.5" x 7.5" are 7.1"x 4"x 2" you ..and rest of our conversation'
mod_str = re.sub(r'(?<=[\dx])["\s]+(?=[x\s])|(?<=x)\s(?=\d)', '', my_str)
print(mod_str)
gets you:
'1.5x3x10 hey 7x4x2 how 9.5x9.5x7.5 are 7.1x4x2 you ..and rest of our conversation
It would probably be faster (and easier to capture outliers) if you were to split this into a multi-step process.
Explanation:
There are two search patterns here, (?<=[\dx])["\s]+(?=[x\s]) and (?<=x)\s(?=\d), they are separated by | to denote one or the other (in left-to-right fashion, so if the first group captures a piece of content the second won't be executed on it).
The first:
(?<= positive non-capturing lookbehind, capture the next segment only if match
[\dx]) match a single digit (0-9) or the 'x' character
)
["\s]+ match one or more " characters or whitespace
(?= positive non-capturing lookahead, capture the previous segment only if match
[x\s] match a single whitespace or 'x' character
)
The second:
(?<= positive non-capturing lookbehind, capture the next segment only if match
x match the 'x' character
)
\s match a single whitespace
(?= positive non-capturing lookahead, capture the previous segment only if match
\d match a single digit (0-9)
)
The first takes care of selecting whitespace and quotation marks around your digits, the second extends selecting white space around "x" characters only if followed by number to augment the deficiency of the first pattern. Together, they match the correct quotation marks and whitespaces which then get replaced by empty string using the re.sub() method.