Django `bulk_create` with related objects

Question

I have a Django system that runs billing for thousands of customers on a regular basis. Here are my models:

class Invoice(models.Model):
    balance = models.DecimalField(
        max_digits=6,
        decimal_places=2,
    )

class Transaction(models.Model):
    amount = models.DecimalField(
        max_digits=6,
        decimal_places=2,
    )
    invoice = models.ForeignKey(
        Invoice,
        on_delete=models.CASCADE,
        related_name='invoices',
        null=False
    )

When billing is run, thousands of invoices with tens of transactions each are created using several nested for loops, which triggers an insert for each created record. I could run bulk_create() on the transactions for each individual invoice, but this still results in thousands of calls to bulk_create().

How would one bulk-create thousands of related models so that the relationship is maintained and the database is used in the most efficient way possible?

Notes:

I'm looking for a native Django solution that would work on all databases (with the possible exception of SQLite).
My system runs billing in a celery task to decouple long-running code from active requests, but I am still concerned with how long it takes to complete a billing cycle.
The solution should assume that other requests or running tasks are also reading from and writing to the tables in question.

Community · Accepted Answer · 2017-05-23 11:52:59Z

15

You could bulk_create all the Invoice objects, refresh them from the db, so that they all have ids, create the Transaction objects for all the invoices and then also save them with bulk_create. All of this can be done inside a single transaction.atomic context.

Also, specifically for django 1.10 and postrgres, look at this answer.

edited May 23, 2017 at 11:52

CommunityBot

11 silver badge

answered Nov 24, 2016 at 16:45

Ivan

6,0432 gold badges32 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sagar Adhikari · Accepted Answer · 2022-05-09 10:49:25Z

4

You can do it with two bulk create queries, with following method.

new_invoices = []
new_transactions = []
for loop:
    invoice = Invoice(params)
    new_invoices.append(invoice)

    for loop: 
        transaction = Transaction(params)
        transaction.invoice = invoice
        new_transactions.append(transaction)

Invoice.objects.bulk_create(new_invoices)

for each in new_transactions:
    each.invoice_id = each.invoice.id

Transaction.objects.bulk_create(new_transactions)

edited May 9, 2022 at 10:49

answered May 9, 2022 at 10:41

Sagar Adhikari

1,4021 gold badge14 silver badges19 bronze badges

2 Comments

shawnngtq Over a year ago

I know official doc says save / pre_save / post_save method won't be called, is it the same case for this answer? I need save method called as I customize actions

Sagar Adhikari Over a year ago

Yes. save method wont be called.

Javad · Accepted Answer · 2022-11-21 10:36:41Z

1

Another way for this purpose can be like the below code snippet:

from django.utils import timezone
from django.db import transaction

new_invoices = []
new_transactions = []
for sth in sth_else:
    ...
    invoice = Invoice(params)
    new_invoices.append(invoice)

for sth in sth_else:
    ...
    new_transactions.append(transaction)

with transaction.atomic():
    other_invoice_ids = Invoice.objects.values_list('id', flat=True)
    now = timezone.now()
    Invoice.objects.bulk_create(new_invoices)

    new_invoices = Invoice.objects.exclude(id__in=other_invoice_ids).values_list('id', flat=True)
    for invoice_id in new_invoices:
                transaction = Transaction(params, invoice_id=invoice_id)
                new_transactions.append(transaction)

    Transaction.objects.bulk_create(new_transactions)

I write this answer based on this post on another question in the community.

answered Nov 21, 2022 at 10:36

Javad

2,1333 gold badges16 silver badges25 bronze badges

2 Comments

shawnngtq Over a year ago

I know official doc says save / pre_save / post_save method won't be called, is it the same case for this answer? I need save method called as I customize actions

Javad Over a year ago

bulk_create won't call save() method, if you need save() method you can create a bulk_create() for your manager or a new method do bulk create.

Collectives™ on Stack Overflow

Django `bulk_create` with related objects

3 Answers 3

Comments

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related