Let's break this down as an attempt (a rough one) to explore why the compiler issues the error.
1. The difference between andThen and compose in this context
Here are the signatures of the two methods (omitted default):
<V> Function<T, V> andThen(Function<? super R, ? extends V> after)
<V> Function<V, R> compose(Function<? super V, ? extends T> before)
Remembering that Function is declared with two type variables, <T, R>, the most significant difference between after and before in this context is that after is expected to take in a parameter whose data type is R, the current function's return type, whereas the data type of before's parameter is compose's local type variable, V, which the compiler must infer.
Why is this relevant?
2. Why does test.andThen(String::toUpperCase) work?
Among the many things that the compiler is trying to do, it's trying to validate that String::toUpperCase is a valid argument to andThen and compose. To do that, it needs to establish the type of String::toUpperCase, meaning it needs to infer the data type of the method reference in that invocation context. Well, the compiler knows that String::toUpperCase must be a Function, so it just needs to infer the type arguments for Function, which is where the difference above applies:
- In the case of
after (the parameter of andThen), the first parameter of Function<? super R, ? extends V>, is known to be (roughly) R, which has to be the same as the current function's (test's) return type. And the second type parameter, V, which also corresponds to andThen's local type variable, is inferred to whatever the method reference will be declared to return.
- In the case of
before (the parameter of compose), the first parameter's type is unknown, and has to be inferred in context. It corresponds to compose's type variable V. This is where the first difference is.
In Eclipse, the compiler error was The type String does not define toUpperCase(Object) that is applicable here, which gives the clue that the error is related to matching the argument type for String#toUpperCase.
Now, if you see above:
after is a Function that takes a String parameter, which makes it somehow simple for the compiler to resolve the String method that String::toUpperCase is targeting. This is a massive topic on its own, but it boils down to the compiler choosing to link the function roughly as the lambda expression (String s) -> s.toUpperCase(). Method references can resolve to instance methods like that, where the first parameter of the function becomes the target of the method invocation (without parameters in the case of Function<T, U>).
before could have been resolved in the same way, except that the compiler does NOT know for sure what the type of Function's first type argument is. And because String#toUpperCase is overloaded, the compiler can't just decide to infer it as (String s) -> s.toUpperCase(), because V could have been meant to be of a data type that would force it to resolve the method reference to the overload, String.toUpperCase(Locale). In other words, the compiler is dealing with a chicken-and-the-egg situation (where it needs to know the parameter type V to decide which String#toUpperCase method to link, but can't use String#toUpperCase signature to infer V, because there are two possibilities making it ambiguous).
3. How you solved it
You got this code to compile by casting String::toUpperCase to Function<String, String>. What did that do? It simply got the compiler out of the above dilemma: you told the compiler that V is String, not Locale or any other possibility, which caused the compiler to resolve the method reference as (String s) -> s.toUpperCase() (leaving no room for String.toUpperCase(Locale) to be a valid option)
4. Other ways to solve it
You can effectively do the same thing by forcing an explicit type argument for before's V:
test.compose((String s) -> s.toUpperCase()).apply("bar"); will compile because (String s) explicitly types the data type of before's parameter
test.<String>compose(String::toUpperCase).apply("bar"); does the same thing by helping the compiler's inference logic, telling it that V is a String, which avoids the chicken-and-the-egg situation referred to above.
This type of solution is not some strange approach, you're essentially doing the same thing that the cast in System.out.println((String)null) does when System.out.println(null) is rejected by the compiler, although these Function methods include generics as a spice.