Algorithm to list unique permutations of string with duplicate letters

This is not full answer, just an idea.

If your strings has fixed number of only two letters I'll go with binary tree and good recursion function. Each node is object that contains name with prefix of parent name and suffix A or B furthermore it have numbers of A and B letters in the name.

Node constructor gets name of parent and number of A and B from parent so it needs only to add 1 to number of A or B and one letter to name.

It doesn't construct next node if there is more than three A (in case of A node) or B respectively, or their sum is equal to the length of starting string.

Now you can collect leafs of 2 trees (their names) and have all permutations that you need.

Scala or some functional language (with object-like features) would be perfect for implementing this algorithm. Hope this helps or just sparks some ideas.

For a multiset, you can solve recursively by position (JavaScript code):

function f(multiset,counters,result){
  if (counters.every(x => x === 0)){
    console.log(result);
    return;
  }

  for (var i=0; i<counters.length; i++){
    if (counters[i] > 0){
      _counters = counters.slice();
      _counters[i]--;
      f(multiset,_counters,result + multiset[i]);
    }
  }
}

f(['A','B'],[3,3],'');

Since you actually want to generate the permutations instead of just counting them, the best complexity you can hope for is O(size_of_output).

Here's a good solution in java that meets that bound and runs very quickly, while consuming negligible space. It first sorts the letters to find the lexographically smallest permutation, and then generates all permutations in lexographic order.

It's known as the Pandita algorithm: https://en.wikipedia.org/wiki/Permutation#Generation_in_lexicographic_order

import java.util.Arrays;
import java.util.function.Consumer;


public class UniquePermutations
{
    static void generateUniquePermutations(String s, Consumer<String> consumer)
    {
        char[] array = s.toCharArray();
        Arrays.sort(array);
        for (;;)
        {
            consumer.accept(String.valueOf(array));

            int changePos=array.length-2;
            while (changePos>=0 && array[changePos]>=array[changePos+1])
                --changePos;

            if (changePos<0)
                break; //all done

            int swapPos=changePos+1;
            while(swapPos+1 < array.length && array[swapPos+1]>array[changePos])
                ++swapPos;

            char t = array[changePos];
            array[changePos] = array[swapPos];
            array[swapPos] = t;

            for (int i=changePos+1, j = array.length-1; i < j; ++i,--j)
            {
                t = array[i];
                array[i] = array[j];
                array[j] = t;
            }
        }
    }

    public static void main (String[] args) throws java.lang.Exception
    {
        StringBuilder line = new StringBuilder();
        generateUniquePermutations("banana", s->{
            if (line.length() > 0)
            {
                if (line.length() + s.length() >= 75)
                {
                    System.out.println(line.toString());
                    line.setLength(0);
                }
                else
                    line.append(" ");
            }
            line.append(s);
        });
        System.out.println(line);
    }
}

Here is the output:

aaabnn aaanbn aaannb aabann aabnan aabnna aanabn aananb aanban aanbna
aannab aannba abaann abanan abanna abnaan abnana abnnaa anaabn anaanb
anaban anabna ananab ananba anbaan anbana anbnaa annaab annaba annbaa
baaann baanan baanna banaan banana bannaa bnaaan bnaana bnanaa bnnaaa
naaabn naaanb naaban naabna naanab naanba nabaan nabana nabnaa nanaab
nanaba nanbaa nbaaan nbaana nbanaa nbnaaa nnaaab nnaaba nnabaa nnbaaa