.+ is greedy. It takes as much as it can. That is half of the has so that \1 can match the second half. Making the repetition ungreedy should do the trick:
/[^\w\s]|(.+?)\1+/gi
By the way, the i doesn't change anything here.
To get rid of nested repetitions (e.g. transform aaBBaaBB into aB (via aaBB or aBaB)) simply run the replacement multiple times until the result does not change any more.
var pattern = /[^\w\s]|(.+?)\1+/g;
var output = "aaBBaaBB";
var input;
do
{
input = output;
output = input.replace(pattern, "$1");
} while (input != output)
I admit the naming of output is a bit awkward for the first repetition, but you know... the two most difficult problems in computer science are cache invalidation, naming things and off-by-one errors.
abcabcDabcabcD. Do you wantabcDabcDorabcabcD? So longest, or shortest repetition if there are nested repetitions.