String::toUpperCase is a method reference that can be applied to any String instance.
new String()::toUpperCase is a method reference that can be applied to a specific String instance (the instance created by new String()).
Since UnaryOperator<String> expects a method that takes a String and returns a String, String::toUpperCase fits (since you can apply it on a String and get the upper case version of that String).
On the other hand, new String()::toUpperCase doesn't fit UnaryOperator<String>, since it is executed on an already specified String, so you can't pass another String instance to it.
It can, however, by assigned to a Supplier<String>, since it simply supplies an empty String instance:
Supplier<String> emptyStringToUpperCase = new String()::toUpperCase;
This is similar to:
Supplier<String> emptyStringToUpperCase = () -> new String().toUpperCase();
while this:
UnaryOperator<String> stringToUpperCase = String::toUpperCase;
is similar to:
UnaryOperator<String> stringToUpperCase = s -> s.toUpperCase();
s -> ""will do the same thing