There is no way, how your code
List<Integer> list = map.values().stream()
.map(m -> m.entrySet().stream()
.map(e -> e.getKey())
.collect(Collectors.toList()))
.flatMap(l -> l.stream())
.collect(Collectors.toList());
can produce a ClassCastException, unless you managed to insert objects of wrong type into the source map via unchecked operation(s) before the Stream operation. Such a situation is called heap pollution and you should compile your entire code with all warnings enabled (javac: use option -Xlint:unchecked) and solve them.
But note that your code is unnecessarily complicated. The chain, .entrySet().stream().map(e -> e.getKey()) is streaming over the entries and mapping to the keys, so you can stream over the keys in the first place, i.e. .keySet().stream(). Then, you are collecting the stream into a List, just to invoke .stream() in the subequent flatMap step, so you can simply use the stream you already have instead:
List<Integer> list = map.values().stream()
.flatMap(m -> m.keySet().stream())
.collect(Collectors.toList());
Alternatively, you can let the collector do all the work:
List<Integer> list = map.values().stream()
.collect(ArrayList::new, (l,m) -> l.addAll(m.keySet()), List::addAll);
Getting the values instead of the keys works similar, but requires another flatMap step to get the List elements:
List<Integer> list = map.values().stream()
.flatMap(m -> m.values().stream().flatMap(List::stream))
.collect(Collectors.toList());
which is equivalent to
List<Integer> list = map.values().stream()
.flatMap(m -> m.values().stream())
.flatMap(List::stream)
.collect(Collectors.toList());
Again, there’s the alternative of letting the collector do all the work:
List<Integer> list = map.values().stream()
.collect(ArrayList::new, (l,m)->m.values().forEach(l::addAll), List::addAll);
or
List<Integer> list = map.values().stream()
.collect(ArrayList::new, (l,m)->m.forEach((k,v)->l.addAll(v)), List::addAll);