Java Collection with generic method and subclasses

The intent is that for any class X that extends TaskCollection, when a groupBy operation is performed, the collection used for the map values are also instances of class X.

In that case, the closest you can get to that is something like the following:

class Task {}

class Assertion extends Task {}

abstract class TaskCollection<E extends Task, C extends TaskCollection<E, C>> extends HashSet<E> {

    <K> Map<K, C> groupBy(Function<E, K> groupingFunction) {
        return this.stream()
            .collect(Collectors.groupingBy(
                    groupingFunction,
                    Collectors.toCollection(this.collectionSupplier())
            ));
    }

    protected abstract Supplier<C> collectionSupplier();
}

class AssertionCollection extends TaskCollection<Assertion, AssertionCollection> {

    @Override
    protected Supplier<AssertionCollection> collectionSupplier() {
        return AssertionCollection::new;
    }
}

Notice that the definition of TaskCollection above does not quite stop subclasses of using another TaskCollection class for their groupBy map values. For example this would also compile:

class AssertionCollectionOther extends TaskCollection<Assertion, AssertionCollectionOther> {...}

class AssertionCollection extends TaskCollection<Assertion, AssertionCollectionOther> {...}

Unfortunately it is not possible to impose such a constraint, at least for now, as you cannot make reference to the class that is being declared in the C type-parameter wildcard.

If you can assume that descendants have a parameter free constructor as the collection supplier you can provide a default implementation for collectionSupplier. The price you pay is the need to silence a "unchecked" warning (not a real problem) and that not compliant classes (not providing the parameter-free constructor) won't fail at compilation time but at run-time which is less ideal:

import java.util.function.*;
import java.util.*;
import java.util.stream.*;

class Task {}

class Assertion extends Task {}

class TaskCollection<E extends Task, C extends TaskCollection<E, C>> extends HashSet<E> {

    <K> Map<K, C> groupBy(Function<E, K> groupingFunction) {
        return this.stream()
            .collect(Collectors.groupingBy(
                    groupingFunction,
                    Collectors.toCollection(this.collectionSupplier())
            ));
    }

    @SuppressWarnings("unchecked")
    protected Supplier<C> collectionSupplier() {
       return () -> {
         try {
          return (C) this.getClass().newInstance();
         } catch (Exception ex) {
          throw new RuntimeException(String.format("class %s is not a proper TaskCollection", this.getClass()), ex);
         }
      };
    }
}

class AssertionCollection extends TaskCollection<Assertion, AssertionCollection> {
    // This override is not needed any longer although still could
    // be included in order to produce a slightly faster 
    // customized implementation:
    //@Override
    //protected Supplier<AssertionCollection> collectionSupplier() {
    //    return AssertionCollection::new;
    //}
}

If you declare collectionSupplier as final you would effectively force subclasses to always return instances of their own class with the caveat that a, then non-sense, declaration such as class AssertionCollection extends TaskCollection<Assertion, AssertionCollectionOther> would still compile and produce run-time cast exceptions down the road.