Download csv file using python 3

Question

I am new to Python. Here is my environment setup:

I have Anaconda 3 ( Python 3). I would like to be able to download an CSV file from the website: https://data.baltimorecity.gov/api/views/dz54-2aru/rows.csv?accessType=DOWNLOAD

I would like to use the requests library. I would appreciate anyhelp in figuring our how I can use the requests library in downloading the CSV file to the local directory on my machine

Have you found a better solution? I believe that you can adjust the size of the buffer for a better fetch performance. — Andre Pastore
– Andre Pastore, Commented Oct 23, 2015 at 18:27

Community · Accepted Answer · 2017-05-23 11:44:46Z

1

It is recommended to download data as stream, and flush it into the target or intermediate local file.

import requests


def download_file(url, output_file, compressed=True):
    """
    compressed: enable response compression support
    """
    # NOTE the stream=True parameter. It enable a more optimized and buffer support for data loading.
    headers = {}
    if compressed:
        headers["Accept-Encoding"] = "gzip"

    r = requests.get(url, headers=headers, stream=True)

    with open(output_file, 'wb') as f: #open as block write.
        for chunk in r.iter_content(chunk_size=4096): 
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)
        f.flush() #Afterall, force data flush into output file (optional)

    return output_file

Considering original post:

remote_csv = "https://data.baltimorecity.gov/api/views/dz54-2aru/rows.csv?accessType=DOWNLOAD"
local_output_file = "test.csv"

download_file(remote_csv, local_output_file)

#Check file content, just for test purposes:
print(open(local_output_file).read())

Base code was extracted from this post: https://stackoverflow.com/a/16696317/176765

Here, you can have more detailed information about body stream usage with requests lib:

http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow

edited May 23, 2017 at 11:44

CommunityBot

11 silver badge

answered Oct 19, 2015 at 0:44

Andre Pastore

2,9074 gold badges35 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user3049935 Over a year ago

Thanks here is the code I used. This might sound basic and would appreciate any changes/updates to make it better: import requests r = requests.get("data.baltimorecity.gov/api/views/dz54-2aru/…) with open('test.csv', 'wb') as f: f.write(r.content)

Andre Pastore Over a year ago

Consider edited post as a general download purpose code. You can use it with any content format.

Collectives™ on Stack Overflow

Download csv file using python 3

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related