let f a b c = a (a b c) 0;;
Your confusion involves types and type inference, i.e., when you define a function or binding, you don't need to give explicit types for its parameters, nor the function/binding itself, OCaml will figure it out if your definition of function/binding is correct.
So, let's do some manual inferences ourselves. If a human can do, then the compiler can also do.
1.
let x = 1
1 is integer, so x must be an integer. So you don't need to do int x = 1 as in other languages, right?
2.
let f x = 1
If there are multiple variable names between let and =, then it must be a function definition, right? Otherwise, it won't make sense. In Java like language, it also makes no sense to say int x y = 1, right?
So f is a function and x is must be a parameter. Since the righthand side of = is an integer, then we know f will return an integer. For x, we don't know, so x will be thought as a polymorphic type 'a.
So f: 'a -> int = <fun>.
3.
let f a b c = a (a b c) 0;;
f is a function, with parameters a, b, c.
a must be a function, because on the righthand side of =, it is a function application.
a takes two arguments: (a b c) and 0. 0 is an integer, so the 2nd parameter of a must be an integer type.
Look inside (a b c), c is the 2nd arguement, so c must be integer.
We can't infer on b. So b is 'a.
Since (a b c) can be the 1st argument of a, and (a b c) itself is an application on function a, the return type of a will have the same type of b which is 'a.
Combine information above together, you get f: ('a -> int -> 'a) -> 'a -> int -> 'a.
If you want to learn it formally, https://realworldocaml.org/ is your friend.