There is some underlying structure that can be used to solve this problem in O(n) time.
The rules given are (most of) the rules defining a mathematical group, in particular the group D_2 also sometimes known as K (for Klein's four group) or V (German for Viergruppe, four group). D_2 is a group with four elements, A, B, C, and 1 (the identity element). One of the realizations of D_2 is the set of symmetries of a rectangular box with three different sides. A, B, and C are 180 degree rotations about each of the axes, and 1 is the identity rotation (no rotation). The group table for D_2 is
|1 A B C
-+-------
1|1 A B C
A|A 1 C B
B|B C 1 A
C|C B A 1
As you can see, the rules correspond to the rules given in the problem, except that the rules involving 1 aren't present in the problem.
Since D_2 is a group, it satisfies a number of rules: closure (the product of any two elements of the group is another element), associativity (meaning (x*y)*z = x*(y*z) for any elements x, y, z; i.e., the order in which strings are reduced doesn't matter), existence of identity (there is an element 1 such that 1*x=x*1=x for any x), and existence of inverse (for any element x, there is an element x^{-1} such that x*x^{-1}=1 and x^{-1}*x=1; in our case, every element is its own inverse).
It's also worth noting that D_2 is commutative, i.e., x*y=y*x for any x,y.
Given any string of elements in D_2, we can reduce to a single element in the group in a greedy fashion. For example, ABCCCCCCC=CCCCCCCC=CCCCCC=CCCC=CC=1. Note that we don't write the element 1 unless it's the only element in the string. Associativity tells us that the order of the operations doesn't matter, e.g., we could have worked from right to left or started in the middle and gotten the same result. Let's try from the right: ABCCCCCCC=ABCCCCC=ABCCC=ABC=AA=1.
The situation of the problem is different because operations involving 1 are not allowed, so we can't just eliminate pairs AA, BB, or CC. However, the situation is not that different. Consider the string ABB. We can't write ABB=A in this case. However, we can eliminate BB in two steps using A: ABB=CB=A. Since order of operation doesn't matter by associativity, we're guaranteed to get the same result. So we can't go straight from ABB to A but we can get the same result by another route.
Such alternate routes are available whenever there are at least two different elements in a string. In particular, in each of ABB, ACC, BAA, BCC, CAA, CBB, AAB, AAC, BBA, BBC, CCA, CCB, we can act as if we have the reduction xx=1 and then drop the 1.
It follows that any string that is not homogeneous (not all the same letter) and has a double-letter substring (AA, BB, or CC) can be reduced by removing the double letter. Strings that contain just two identical letters can't be further reduced (because there is no 1 allowed in the problem), so it seems safe to hypothesize that any non-homogeneous string can be reduced to A, B, C, AA, BB, CC.
We still have to be careful, however, because CCAACC could be turned into CCCC by removing the middle pair AA, but that is not the best we can do: CCAACC=AACC=CC or AA takes us down to a string of length 2.
Another situation we have to be careful of is AABBBB. Here we could eliminate AA to end with BBBB, but it's better to eliminate the middle B's first, then whatever: AABBBB=AABB=AA or BB (both of which are equivalent to 1 in the group, but can't be further reduced in the problem).
There's another interesting situation we could have: AAAABBBB. Blindly eliminating pairs takes us to either AAAA or BBBB, but we could do better: AAAABBBB=AAACBBB=AABBBB=AABB=AA or BB.
The above indicate that eliminating doubles blindly is not necessarily the way to proceed, but nevertheless it was illuminating.
Instead, it seems as if the most important property of a string is non-homogeneity. If the string is homogeneous, stop, there's nothing we can do. Otherwise, identify an operation that preserves the non-homogeneity property if possible. I assert that it is always possible to identify an operation that preserves non-homogeneity if the string is non-homogeneous and of length four or greater.
Proof: if a 4-substring contains two different letters, a third letter can be introduced at a boundary between two different letters, e.g., AABA goes to ACA. Since one or the other of the original letters must be unchanged somewhere within the string, it follows that the result is still non-homogeneous.
Suppose instead we have a 4-substring that has three different elements, say AABC, with the outer two elements different. Then if the middle two elements are different, perform the operation on them; the result is non-homogeneous because the two outermost elements are still different. On the other hand, if the two inner elements are the same, e.g., ABBC, then they have to be different from both outermost elements (otherwise we'd only have two elements in the set of four, not three). In that case, perform either the first or third operation; that leaves either the last two elements different (e.g., ABBC=CBC) or the first two elements different (e.g., ABBC=ABA) so non-homogeneity is preserved.
Finally, consider the case where the first and last elements are the same. Then we have a situation like ABCA. The middle two elements both have to be different from the outer elements, otherwise we'd have only two elements in this case, not three. We can take the first available operation, ABCA=CCA, and non-homogeneity is preserved again.
End of proof.
We have a greedy algorithm to reduce any non-homogeneous string of length 4 or greater: pick the first operation that preserves non-homogeneity; such an operation must exist by the above argument.
We have now reduced to the case where we have a non-homogeneous string of 3 elements. If two are the same, we either have doubles like AAB etc., which we know can be reduced to a single element, or we have two elements with no double like ABA=AC=B which can also be reduced to a single element, or we have three different elements like ABC. There are six permutations, all of which =1 in the group by associativity and commutativity; all of them can be reduced to two elements by any operation; however, they can't possibly be reduced below a homogeneous pair (AA, BB, or CC) since 1 is not allowed in the problem, so we know that's the best we can do in this case.
In summary, if a string is homogeneous, there's nothing we can do; if a string is non-homogeneous and =A in the group, it can be reduced to A in the problem by a greedy algorithm which maintains non-homogeneity at each step; the same if the string =B or =C in the group; finally if a string is non-homogeneous and =1 in the group, it can be reduced by a greedy algorithm which maintains non-homogeneity as long as possible to one of AA, BB or CC. Those are the best we can do by the group properties of the operation.
Program solving the problem:
Now, since we know the possible outcomes, our program can run in O(n) time as follows: if all the letters in the given string are the same, no reduction is possible so just output the length of the string. If the string is non-homogeneous, and is equal to the identity in the group, output the number 2; otherwise output the number 1.
To quickly decide whether an element equals the identity in the group, we use commutativity and associativity as follows: just count the number of A's, B's and C's into the variables a, b, c. Replace a = a mod 2, b = b mod 2, c = c mod 2 because we can eliminate pairs AA, BB, and CC in the group. If none of the resulting a, b, c is equal to 0, we have ABC=1 in the group, so the program should output 2 because a reduction to the identity 1 is not possible. If all three of the resulting a, b, c are equal to 0, we again have the identity (A, B, and C all cancelled themselves out) so we should output 2. Otherwise the string is non-identity and we should output 1.
ABCCCCCCCbecomeAACCCCCC? The rule would suggest that theABbecomesC.BCtoAin this step to obtainAACCCCCC. In other words, it is not required to reduce the leftmost matching substring, you may reduce any substring that consists of two different characters.