How to download large file without MemoryError in Python?

Question

I would like to programmatically download some files but am getting MemoryError exception for larger ones. For example, when I try to download a small file, the code is fine, but when I try to download a larger file, I catch a MemoryError.

Here is my code:

def __download_gpl_file(accession):
    try:
        bin_string = __get_response(accession)
        if bin_string is None:
            return False
        string = __unzip(bin_string)
    except MemoryError:
        print 'Out of memory for: ' + accession
        return False

    if string:
        filename = DOWNLOADED + accession + '.txt'
        with open(filename, 'w+') as f:
            f.write(string)
        return True
    return False


def __get_response(attempts=5):
    url = __construct_gpl_url(accession)  # Not shown
    response = None
    while attempts > 0:
        try:
            response = urllib2.urlopen(url)
            if response and response.getcode() < 201:
                break
            else:
                attempts -= 1
        except urllib2.URLError:
            print 'URLError with: ' + url
    return response.read()


def __unzip(bin_string):
    f = StringIO(bin_string)
    decompressed = gzip.GzipFile(fileobj=f)
    return decompressed.read()

Is there anything I can do to download larger files? Thanks in advance.

Can the downvoter please explain how to improve this question? — jds
– jds, Commented Nov 21, 2014 at 15:02

jds · Accepted Answer · 2017-06-14 15:27:23Z

6

instead of writing whole file at once , you write line by line:

file = urllib2.urlopen('url')
with open('filename','w') as f:
    for x in file:
        f.write(x)

if you want to make it more fast:

file = urllib2.urlopen('url')
with open('filename','w') as f:
    while True:
        tmp = file.read(1024)
        if not tmp:
            break 
        f.write(tmp)

edited Jun 14, 2017 at 15:27

jds

8,31013 gold badges69 silver badges114 bronze badges

answered Nov 21, 2014 at 2:48

Hackaholic

19.8k6 gold badges59 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mark Dickie · Accepted Answer · 2017-06-14 08:53:09Z

2

I don't have enough points to comment on Hackaholic's answer so my answer is just his first example but with a slight correction.

file = urllib2.urlopen('url') 
with open('filename','w') as f:
    for x in file:
        f.write(x)

I think he wrote f.write(f) by accident.

answered Jun 14, 2017 at 8:53

Mark Dickie

214 bronze badges

Collectives™ on Stack Overflow

How to download large file without MemoryError in Python?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related