How to implement a Kotlin Comparator with Reflect and safe cast?

There are (at least) two major problems with your approach.

First of all, it is conceptually wrong. Only because a property is Comparable<*>, doesn't mean we can compare it to the same property in the second object. It can compare to anything, not necessary to itself:

fun main() {
    val o1 = MyClass()
    val o2 = MyClass()
    o1.prop1.compareTo(o2.prop1) // doesn't compile
}

class MyClass {
    val prop1: Comparable<String> = TODO()
}

We would rather need to check the type parameter of the Comparable and verify the property is of the same type. Then we can safely cast. However, such solution will be much more complicated than what you do right now, and it requires to use KType. Also, there are some corner cases to keep in mind, for example if the T of Comparable is erased and we can't easily verify it.

Second, you assume both a and b are of the same type, which is not necessarily true. What if a is a subtype of b and adds extra properties? In that case prop.getter.call(b) will fail for props b lacks. As a matter of fact, the compiler shouldn't allow you to write such code. If you use the "regular" way to get properties, i.e.: prop.get(b), then the code would not compile, saying types are not correct. You workarounded this by using prop.getter.call(b), which ignores types, but that doesn't mean this is safe.

You can add a runtime check that a::class == b::class or maybe compare only properties of the super type - it depends on your case.

Update

Proper implementation using the T of Comparable to detect if the comparison is safe to do, could be something like this:

fun <T:Any> reflectCompare(a: T, b: T) : Int {
    a::class
        .memberProperties
        .intersect(b::class.memberProperties.toSet())
        .forEach { prop ->
            println("Property: $prop")
            @Suppress("UNCHECKED_CAST")
            val pa = (prop as KProperty1<T, *>).get(a)
            val pb = prop.get(b)
            if (pa !is Comparable<*>) {
                println("Not Comparable, ignoring")
                return@forEach
            }

            // T of Comparable
            val type = pa::class.supertypes
                .single { it.classifier == Comparable::class }
                .arguments[0].type!!.classifier
            
            if (type !is KClass<*>) {
                println("T of Comparable is not a class, ignoring")
                return@forEach
            }
            if (!type.isInstance(pb)) {
                println("Value isn't comparable to itself, ignoring")
                return@forEach
            }

            println("Comparing...")
            @Suppress("UNCHECKED_CAST")
            val cmp = (pa as Comparable<Any?>).compareTo(pb)
            if(cmp != 0) return cmp
        }
    return 0
}

Test:

fun main() {
    val comparable = object : Comparable<String> {
        override fun compareTo(other: String) = 0
    }
    val o = MyClass("hello", MyString("world"), comparable, emptyList(), comparable)
    reflectCompare(o, o)
}

class MyClass<T>(
    val prop1: String, // comparable to itself
    val prop2: MyString, // comparable to itself
    val prop3: Comparable<String>, // not comparable to itself
    val prop4: List<Int>, // not comparable to itself
    val prop5: Comparable<T>, // not sure if comparable to itself
)

data class MyString(val s: String) : Comparable<MyString> {
    override fun compareTo(other: MyString) = s.compareTo(other.s)
}

As said above, this is tricky thing to do, there are corner cases and I suspect this implementation doesn't cover all of them. I ignored some cases for simplicity, e.g.: variance. Also, there is no single way to implement this, for example we could use runtime types (as you did and I did as well) or we could use compile types - this could provide a slightly different result.

We probably need more test cases for generic Comparable<T>. prop5 doesn't really test this case as we use the runtime type of prop5, not the compile type.

Anyway, you can use it as a start.