I have a large file to process, a csv file with millions of rows. I upload the file in a Django handler but I can't process it fast enough so I have a separate process handle the parsing. I create a progress url that I can call to get the status of the parsing.
I was trying to test using django.test.client.Client but it seems like SQLite runs as an in-memory database when used inside a django.test.TestCase. This means that some of the data I initialize in my handler process isn't available to the worker process. The test data gets written to the in-memory test database and the worker process can't access the in-memory test database.
I'm feeling that the Django test suite isn't going to be able to handle this test case. Does anyone have a recommendation about how I could go about testing this? Can I get Django to dump the SQLite test database to a file which I can read inside my worker process? Can I test directly against the production database (which in my case will be empty except for data loaded from fixtures before the test begins)?
