Why am I seeing duplicate model objects in Django when filtering? python

Question

file = open('debug.txt', 'w')
db_objects = model.objects.all()

    for i in db_objects:
        db_filtered = model.objects.filter(input_address_db=i.input_address_db)
        f.write('filtered_results = %s' % db_filtered)
        ...perform operations...

here is the output of the file after the first cycle:

db_filtered = [<model: model object>, <model: model object>, <model: model object>, <model: model object>]

I checked ahead though and know that there should be only three but as you can see the debug text shows four.

Two of the above objects are the same one duplicated. I know this because I get an integrity error when I try to transfer the four objects into a new database.

The created_on dates are the primary key so it's also surprising to see the same result twice because I thought an error would be thrown, I now realise that perhaps you might want the same objects repeated in a queryset which is why there is no error but sill I'm having a bit of trouble getting my head around what is actually going on here.

Thanks in advance for any help, it's much appreciated :)

What happens if you change input_address_db=i.input_address_db to input_address_db__exact=i.input_address_db — petkostas
– petkostas, Commented Aug 6, 2014 at 10:04
@gmfreak, I added the model for you, it's a bitcoin app as you can prob tell from the name, it doesnt correspond with the above example but that's just me using better variables for the forum :) — derrend
– derrend, Commented Aug 6, 2014 at 10:09
Everything looks correct. I think answer of @se7entyse7en should solve the problem, but you should use something like django debug-toolbar to debug this as this might some logical loop which might bite you in future — gmfreak
– gmfreak, Commented Aug 6, 2014 at 10:18

se7entyse7en · Accepted Answer · 2014-08-06 11:53:31Z

4

Try this:

db_filtered = model.objects.filter(input_address_db=i.input_address_db).distinct()

Anyway by iterating db_objects as you do, you may iterate over the same input_address_db in more than one iteration. I would do something like:

input_addess_dbs = set(d.input_address_db for d in model.objects.all())
for address in input_address_dbs:
    db_filtered = model.objects.filter(input_address_db=address).distinct()
    ...

edited Aug 6, 2014 at 11:53

answered Aug 6, 2014 at 10:05

se7entyse7en

4,3268 gold badges35 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

derrend Over a year ago

Thanks again, it seems to have done the trick :D I'll let it run for 24 hours and green tick you when all goes well :)

derrend · Accepted Answer · 2014-08-08 00:17:25Z

I appreciate the help with this everybody and your answers helped me a lot, I just want to clarify what the problem ultimately turned out to be -

class bitcoin_deposit(models.Model):
    input_address_db = models.CharField(max_length=34, null=False, blank=False)
    deposit_amount_db = models.DecimalField(max_digits=16, decimal_places=8, validators=[MaxValueValidator(21000000)])
    return_address_db = models.CharField(max_length=34, null=False, blank=False)
    confirmations_db = models.IntegerField(validators=[MinValueValidator(-1)])
    created_on_db = models.DateTimeField(primary_key=True, auto_now_add=True)
    updated_on_db = models.DateTimeField(auto_now=True)
    txlist_db = models.CharField(max_length=64, null=False, blank=False)

I solved the problem by changing txlist_db to the primary key.

Turns out each time the bitcoin network tells your server that you've just received some BTC you actually get the signal about half a dozen times in very very quick succession, so fast in fact that even though I had this model in place -

class bitcoin_deposit_txlist(models.Model):
    txlist_db = models.CharField(max_length=64, null=False, blank=False)

and this as this first line of code in the script with the txlist variable representing the network transaction id -

txlist = ''btc network transaction id''
if bitcoin_deposit_txlist.objects.filter(txlist_db=txlist).exists():
    exit()
else:
    bitcoin_deposit_txlist.objects.create(txlist_db=txlist)

I was still managing to get the same record recorded more than once, amazingly.

Here are the values of some of the deposit objects that helped me solve it -

[
{'return_address_db': u'mpfUAfK8bHi4VejvV4WhjEar3yQvPoX1Yn', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 3, 478489, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 12, 30, 804646, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.14384551'), 'confirmations_db': 1, 'txlist_db': u'7fd82945b5b96ae4ff866b6689867038289b81ee947c8796007d3a9f1fa14b59', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mmgC3dk3JRpCrS9UDS8QRHFokFd6N8bLz4', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 3, 491573, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 44, 784596, tzinfo=<UTC>), 'deposit_amount_db': Decimal('1.13722512'), 'confirmations_db': 4, 'txlist_db': u'3f9ad1a2464ca33513f24f0e59274845b4ea04906ab0df054ef6668160f58cb2', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'motQ8oo7RcgMmXZVCHgmbq5jtkGnxJwa59', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 28, 550850, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 31, 254955, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.42293583'), 'confirmations_db': 3, 'txlist_db': u'b8e7e819c0ab1dbae9aa106c3662e17eb3621749a4f72e8e8f1b991212f05971', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'motQ8oo7RcgMmXZVCHgmbq5jtkGnxJwa59', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 28, 559947, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 31, 165585, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.42293583'), 'confirmations_db': 3, 'txlist_db': u'b8e7e819c0ab1dbae9aa106c3662e17eb3621749a4f72e8e8f1b991212f05971', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mzgA1rXEuV59Zho34o5Lvko6LFpV9hWUq8', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 54, 666174, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 44, 837193, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.11976356'), 'confirmations_db': 4, 'txlist_db': u'adbcfc5b70621bde28ba4f073eca7a12e70533fb57e617ddbce013b4f6bc063b', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mzgA1rXEuV59Zho34o5Lvko6LFpV9hWUq8', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 54, 678403, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 31, 293935, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.11976356'), 'confirmations_db': 3, 'txlist_db': u'adbcfc5b70621bde28ba4f073eca7a12e70533fb57e617ddbce013b4f6bc063b', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mzgA1rXEuV59Zho34o5Lvko6LFpV9hWUq8', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 54, 688175, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 15, 53, 125196, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.11976356'), 'confirmations_db': 2, 'txlist_db': u'adbcfc5b70621bde28ba4f073eca7a12e70533fb57e617ddbce013b4f6bc063b', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}, 

{'return_address_db': u'mzgA1rXEuV59Zho34o5Lvko6LFpV9hWUq8', 'created_on_db': datetime.datetime(2014, 8, 7, 8, 8, 54, 695733, tzinfo=<UTC>), 'updated_on_db': datetime.datetime(2014, 8, 7, 8, 20, 44, 611634, tzinfo=<UTC>), 'deposit_amount_db': Decimal('0.11976356'), 'confirmations_db': 4, 'txlist_db': u'adbcfc5b70621bde28ba4f073eca7a12e70533fb57e617ddbce013b4f6bc063b', 'input_address_db': u'mrcLfESBQqqRG3ZqKhwcjfVgsM4REvt2cL'}
]

Notice how close the created_on_db field values are.

Thanks again :)

Collectives™ on Stack Overflow

Why am I seeing duplicate model objects in Django when filtering? python

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related