A cryptographically secure PRNG (CSPRNG) is usually slower than an ordinary PRNG, but otherwise there's no difference in the randomness it produces. In fact, if there's any practical way you can distinguish the output of a CSPRNG from true randomness, then that's a major flaw in the algorithm or its implementationfor your purposes.
Using the raw values as a permutation map will often make it contain the same number several times, which means it's technically not a permutation map any more. This would make those values appear more often than they should in your Perlin noise. I don't know if this would produce any obvious effect - some wavelets would be going in the same direction, so there might be more regular patterns?
In any case, you can fix this by doing some post-processing to remove duplicate values from the permutation map and replace them with other random values. Or you can try it and see if it affects the quality of your output, which is the important thing.