Struggling with netcore GC and Large Small Lived Arrays

I'm developing an Grammatical Evolution Engine that does the following:

1. Parse a file with the BNF rules.
   <letter> ::= a|b|c|d

2. Generates random solutions based in some specific rules. (Basically, generates int arrays)
   i1 = [22341,123412,521123, 123123], i2 = [213213, 123,5125,634643]

3. Maps those int arrays into the rules in the bnf file:
   i1 = [22341,123412,521123, 123123] => ddbca

4. Checks those solutions with some target previously defined.
   i1 value ('ddbca') is ('hello_world') ? 0, else 1

5. Selects the best performing solutions (top 5, top 10, etc) for latter usage

6. Randomly, picks 2 solutions from the solution list, and perform a crossover:
   i1 = [22341,123412,521123, 123123], i2 = [213213, 123,5125,634643]
i1 x i2 => [22341,123412, 5125,634643]
   

7. Based in some predefined probability, executes a mutation in all individuals:
   

for(int i = 0; i < i3.length; i++)
{
    if(random.NextDouble() <= 0.5) {
        i3[i] = random.Next()
    }
}

8. Again, execute mapping:
   i3 = [22341,123412, 5125,634643] => qwerast

The problem that i'm facing is: My algorithm is generating really large int arrays, but all of then are small lived. After a generation, all solutions that weren't selected, should be disposed. But, since the arrays are getting bigger, almost all of then go to the LOH, and when the GC goes to collect then, my application performance drops drastically.

In a single core environment, it starts at 15 generations/s, and after 160 generations, this drops to 3 generations per second.

I already tried to use ArrayPool, but, since i have hundreds of solutions in memory, i saw no performance improvement, and a great impact on memory usage.

I tried to used the ChunkedList Idea from this link, and the performance did not improve, but the LOH drops considerably.

I already change most of my classes to structs, tried to optimize most simple thing (Avoid Linq, use for insted of foreach, etc), but the big performance hit are in those large arrays.

Any of you can think in some kind of solution for this problem that i'm facing?

Thank you in advance!