Django ORM exclude fails

I have some problems with my query - with filter() it's ok but with exclude() doesn't work. My models:

class Dictionary(DateTimeModel):
    base_word = models.ForeignKey(BaseDictionary, related_name=_('dict_words'))
    word = models.CharField(max_length=64)
    version = models.ForeignKey(Version)

class FrequencyData(DateTimeModel):
    word = models.ForeignKey(Dictionary, related_name=_('frequency_data'))
    count = models.BigIntegerField(null=True, blank=True)
    source = models.ForeignKey(Source, related_name=_('frequency_data'), null=True, blank=True)
    user = models.ForeignKey(settings.AUTH_USER_MODEL, related_name=_('frequency_data'))
    user_ip_address = models.GenericIPAddressField(null=True, blank=True)
    date_of_checking = models.DateTimeField(null=True, blank=True)
    is_checked = models.BooleanField(default=False)

And I want to get some words from Dictionary where whose frequencies are not from some user

Dictionary.objects.prefetch_related('frequency_data').filter(frequency_data__user=1)[:100] - OK

Dictionary.objects.prefetch_related('frequency_data').exclude(frequency_data__user=1)[:100] - processor up to 100% and loading

Without prefetch_related the same. What is wrong with exclude?

EDIT Dictionary db tabel - 120k rows FrequencyData - 160k rows

EDIT2 psql(9.6.6)