You can use a data migration for this. First create an empty migration for your app:
$ python manage.py makemigrations yourappname --empty
In your empty migration, create a function to load your data and add a migrations.RunPython operation. Here's a modified version of the one from the Django documentation on migrations:
from __future__ import unicode_literals
from django.db import migrations
def stream_from_api():
...
def load_data(apps, schema_editor):
# We can't import the Person model directly as it may be a newer
# version than this migration expects. We use the historical version.
Person = apps.get_model('yourappname', 'Person')
for item in stream_from_api():
person = Person(first=item['first'], last=item['last'], age=item['age'])
person.save()
class Migration(migrations.Migration):
dependencies = [('yourappname', '0009_something')]
operations = [migrations.RunPython(load_data)]
If you have a lot of simple data, you might benefit from the bulk-creation methods:
from __future__ import unicode_literals
from django.db import migrations
def stream_from_api():
...
def load_data(apps, schema_editor):
# We can't import the Person model directly as it may be a newer
# version than this migration expects. We use the historical version.
Person = apps.get_model('yourappname', 'Person')
def stream_people():
for item in stream_from_api():
yield Person(first=item['first'], last=item['last'], age=item['age'])
# Adjust (or remove) the batch size depending on your needs.
# You won't be able to use this method if your objects depend on one-another
Person.objects.bulk_create(stream_people(), batch_size=10000)
class Migration(migrations.Migration):
dependencies = [('yourappname', '0009_something')]
operations = [migrations.RunPython(load_data)]
Migrations have the added benefit of being automatically enclosed in a transaction, so you can stop the migration at any time and it won't leave your database in an inconsistent state.
manage.py loaddatajust parses your text file and inserts it into your database. You can do that programatically with an initial migration, like it recommends at the bottom of your linked documentation.RunPythonmigration and create a function that streams the data from your API and inserts it into your database. It will be run automatically when the initial migrations are run and will be resistant to corrupting your database state because migrations utilize transactions automatically.