I've just found that I can disable IPv4IPv6 entirely and that makes the trick for me.
Adding to /etc/sysctl.d/local.conf (source):
# Force IPv6 off
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1
And reloading the configuration:
# sysctl --system