You're going to need to stop using a CharacterController and start using a RigidBody. The CharacterController a class directly in the physics engine (PhysX) Because of this, Unity doesn't really allow for much customization on it (and I'm not really sure if they could) and so you lose a lot of control that you get with a straight up RigidBody. What you gain in the CharacterController in simplicity you lose in configurability. Continueing to use CharacterController means constantly fighting the defaults they have put in place to try to do things that are outside of the scope of that class.

When you switch to a RigidBody, consider the following things:

• Use AddForce (or a similar function that suits your use case) for gravity and it will automatically work with collisions
• Preventing the character from tumbling instead of sliding is going to require that you properly set the mass of the character as well as a bit of clever rotation management. This includes locking the rotation to the terrain you're currently on or applying rotational acceleration so the character doesn't flip over. A more complex solution will result in more realism/less seemingly like you're glued to the ground.
• Checking if your grounded is probably the hardest part because there's no one solution to this (and in itself it's kind of vague, like, are you grounded if you're on the ground but upside down?). You'll just have to use some clever raycasting and sweeping to determine this one or maybe a geometric solution.

You're going to need to stop using a CharacterController and start using a RigidBody. The CharacterController is a class directly in the physics engine (PhysX). Because of this, Unity doesn't really allow for much customization on it (and I'm not really sure if they could) and so you lose a lot of control that you get with a straight up RigidBody. What you gain in the CharacterController in simplicity you lose in configurability. Continueing to use CharacterController means constantly fighting the defaults they have put in place to try to do things that are outside of the scope of that class.

When you switch to a RigidBody, consider the following things:

• Use AddForce (or a similar function that suits your use case) for gravity and it will automatically work with collisions
• Preventing the character from tumbling instead of sliding is going to require that you properly set the mass of the character as well as a bit of clever rotation management. This includes locking the rotation to the terrain you're currently on or applying rotational acceleration so the character doesn't flip over. A more complex solution will result in more realism/less seemingly like you're glued to the ground.
• Checking if your grounded is probably the hardest part because there's no one solution to this (and in itself it's kind of vague, like, are you grounded if you're on the ground but upside down?). You'll just have to use some clever raycasting and sweeping to determine this one or maybe a geometric solution.