1

I need to use a secondary SQLite database in a new Django project. This database is on the local filesystem but outside the Django folder. Its path is specified in a .env file at the root of the Django project.

I want Django to be able to manage migrations on that database, but I already have data in it, which I don't want to loose.

I was able to integrate the database into the Django project, and I see no error at any point. I can fetch data from the database via the Django shell. However, when I try to apply migrations, nothing happens: the database is not modified, but Django doesn't give me any error (in fact it says the migration has been applied).

Here's what I did:

  1. created an "archiver" app within Django
  2. within this app, created a routers.py file with the following code:
class ArchiverDbRouter:

    def db_for_read(self, model, **hints):
        if model._meta.app_label in ['archiver']:
            return 'archiver'
        return None

    def db_for_write(self, model, **hints):
        if model._meta.app_label in ['archiver']:
            return 'archiver'
        return None

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        if app_label in ['archiver']:
            return db == 'archiver'
        return None

  1. configured settings.py to use two databases. The idea is to keep the default database for everything Django, and then the "archiver" database for the "archiver" app.
import os
from pathlib import Path
from dotenv import load_dotenv

load_dotenv()
USER_DIR = Path(os.getenv('USER_DIR', './user'))

(...)

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': BASE_DIR / 'db.sqlite3',
    },
    'archiver': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': USER_DIR / 'data/app.db',
    }
}

DATABASE_ROUTERS = ['archiver.routers.ArchiverDbRouter']
  1. generated my models with the inspectdb command:
python manage.py inspectdb --database archiver > tempmodels.py

Then tweaked those models and saved them to archiver/models.py. I removed the managed = False properties in all models.

  1. initialized the database
python manage.py migrate

This creates the "default" database file.

  1. generated the migrations for archiver
python manage.py makemigrations archiver

The 0001_initial.py migration file is created.

  1. applied this migration with the --fake flag
python manage.py migrate archiver 0001 --fake

I can see the corresponding migration saved in the django_migrations table.

At this point, I can use the Django shell and access the actual data in my "archiver" database, which seems to confirm that the routing works correctly and the database is found by Django. E.g.

>>> q = State(account="test", name="test", value="test")
>>> q.save()

Then I see that the new line (with the three "test" values) is present in the "states" table of m "archiver" database (using a third-party tool, HeidiSQL). I can also see that the modified date for the database file has been updated.

  1. made some changes to my models.py, by removing a field that was never used in the Post model.

  2. generated the migrations again

python manage.py makemigrations archiver

The migration file is created:

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('archiver', '0001_initial'),
    ]

    operations = [
        migrations.RemoveField(
            model_name='post',
            name='note',
        ),
    ]
  1. applied the new migration
python manage.py migrate archiver

This gives me the output:

Operations to perform:
  Apply all migrations: archiver
Running migrations:
  Applying archiver.0002_remove_post_note... OK

No error. I can see the corresponding migration saved in the django_migrations table.

HOWEVER, when I explore the archiver database (using HeidiSQL again), the "note" field is still present. Also, the "modified date" for the database file has NOT changed.

What am I missing?

5
  • 1
    When using SQLite, the first thing I would check is whether a new database file has been created in an unexpected location. Commented Sep 23 at 15:10
  • I suspect that this is a router issue. Are you sure your DATABASES connected to the right archiver? For example can you fetch data from the archiver database? Commented Sep 23 at 15:13
  • @willeM_VanOnsem Yes, I am able to read and write to the database using the django shell (as noted in step 7 in the question). Commented Sep 23 at 16:23
  • @snakecharmerb I checked in the Django project root folder but I don't see anything, besides the "default" database, which doesn't contain my tables. That being said, since the first migration for my app was not actually performed (--fake flag), shouldn't the second one should trigger an error? Commented Sep 23 at 16:24
  • Found the problem: the router works (as far as I can tell), but is apparently ignored by the migrate command. See my answer for more info. Commented Sep 23 at 16:55

1 Answer 1

2

So, upon further testing, I noticed that:

Running python manage.py sqlmigrate archiver 0002 --database=archiver shows the correct SQL output:

BEGIN;
--
-- Remove field note from post
--
ALTER TABLE "posts" DROP COLUMN "note";
COMMIT;

Whereas if I don't use the --database flag, then the SQL code is basically empty:

$ python manage.py sqlmigrate archiver 0002
BEGIN;
--
-- Remove field note from post
--
-- (no-op)
COMMIT;

Apparently Django tries to apply the migration to the default database, which results in a chunk of empty SQL code, which obviously has no effect (yet doesn't trigger an error either). Not easy to notice.

So it appears that the router isn't taken into account when using the migrate command, and the solution is to simply be explicit with the target database (if not using the default database):

python manage.py migrate archiver --database=archiver
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.