How can i upload as 1cr rows .csv file using django?

Question

I am getting a memory error while uploading a CSV file of size around 650 mb with a shape (10882101, 6). How can i upload such file in postgres using django framework.

We are going to need more information than this. Please provide the entire error output, as well as the the relevant code. See How to Ask, help center. — AMC
– AMC, Commented May 8, 2020 at 16:03

Anubhav Ujjawal · Accepted Answer · 2020-05-08 15:29:13Z

0

You haven't shared much details (error logs, which python package you are using etc).
You might like to read Most efficient way to parse a large .csv in python?, https://medium.com/casual-inference/the-most-time-efficient-ways-to-import-csv-data-in-python-cc159b44063d
How I will do it from Django Framework:
- I will use Celery to run the job as a background process, as waiting for the file to be uploaded completely before returning a response might give a HTTP timeout.
- Celery quickstart with Django

answered May 8, 2020 at 15:29

Anubhav Ujjawal

928 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can i upload as 1cr rows .csv file using django?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related