Django large CSV export with celery (async)

Question

I am trying to export a CSV file which takes a while to generate. I have written the export CSV view which is successfully working, however I needed to adapt it to use celery for it to work in production with large data volumes. I have written the following code (mostly following https://thoslin.github.io/async-download-with-celery/):

View:

from tasks import export_report
@staff_member_required()
def export_status_report(request):
    task = export_report.delay()
    return render(request, "admin/poll_for_download.html", {"task_id": task.task_id })

# Check if task is finished
def poll_for_download(request):
    task_id = request.GET.get("task_id")
    filename = request.GET.get("filename")

    if request.is_ajax():
        result = AsyncResult(task_id)
        if result.ready():
            return HttpResponse(json.dumps({"filename": result.get()}))
        return HttpResponse(json.dumps({"filename": None}))

    try:
        f = open("/path/to/export/"+filename)
    except:
        return HttpResponseForbidden()
    else:
        response = HttpResponse(file, mimetype='text/csv')
        response['Content-Disposition'] = 'attachment; filename=%s' % filename
    return response

Task:

@app.task
def export_report():
    date = datetime.now()
    date = date.strftime("%m/%d/%Y")
    filename = "reoport"+date+".csv"
    response = HttpResponse(content_type='text/csv')
    response['Content-Disposition'] = 'attachment; filename= "{}"'.format(filename)
    payouts = Payout.objects.all()

    writer = csv.writer(response) #instantiate writer

    # write headers
    writer.writerow(['Field1', 'Field2', 'Field3'])

    for payout in payouts:
        list = [payout.field1, payout.field2, payout.field3]
        writer.writerow(list)

    return filename

HTML

<script>
$(function(){
    $.ajaxSetup({ cache: false, timeout: 360000 });
    var url = "admin/poll-for-download/";
    var i = 0;
    (function worker() {
        $.getJSON(url+"?task_id=", function(data){
            if(data.filename) {
                var file_url = url+"?filename="+data.filename;
                $("#content").html("If your download doesn't start automatically, please click <a href='"+file_url+"'>here</a>.");
                window.location.href = file_url;
            } else {
                setTimeout(worker, 5000);
            }
        });
    })();
    setInterval(function() {
        i = ++i % 4;
        $("#loading").html("loading"+Array(i+1).join("."));
    }, 1000);
});

However, this is not working. I receive an error:

with open("%s%s" % ("/path/to/export/", filename), "w+") as f:
2022-11-10T14:03:49.564942+00:00 app[worker.1]: FileNotFoundError: [Errno 2] No such file or directory: '/path/to/export/31e39843-421d-49f5-8a59-46394c22a3ce.csv'

How do I fix this so that the script will automatically download the CSV when ready and present a link to the user, in case the download does not start automatically. I also don't want the reports to be publicly available, so they should not be uploaded to our S3/Cloudfront static folder.

Yusuf Adel · Accepted Answer · 2023-02-13 14:11:11Z

0

replace file

Task response = HttpResponse(file, mimetype='text/csv')

with

response = HttpResponse(filename, mimetype='text/csv') or f file object

answered Feb 13, 2023 at 14:11

Yusuf Adel

506 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Django large CSV export with celery (async)

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related