0

I have some links in a database which I want to download parallely. I tried doing it serially but it took too much time. I have around 1877 links.

I tried this code for running the downloads parallely but it throws an error: failed: 'tuple' object has no attribute 'read'

#!/usr/bin/env python

import urllib
from stream import ThreadPool

URLs = [
  'http://www.cnn.com/',
  'http://www.bbc.co.uk/',
  'http://www.economist.com/',
  'http://nonexistant.website.at.baddomain/',
  'http://slashdot.org/',
  'http://reddit.com/',
  'http://news.ycombinator.com/'
 ]

def retrieve(urls):
    for url in urls:
    print url,' '
    res = urllib.urlretrieve(url).read()
    yield url, res

if __name__ == '__main__':
    retrieved = URLs >> ThreadPool(retrieve, poolsize=7)
    for url, content in retrieved:
        print '%r is %d bytes' % (url, len(content))
    for url, exception in retrieved.failure:
        print '%r failed: %s' % (url, exception)

I tried this as well:

import urllib
import tldextract
from multiprocessing.pool import ThreadPool

URLs = [
  'http://www.cnn.com/',
  'http://www.bbc.co.uk/',
  'http://www.economist.com/',
  'http://nonexistant.website.at.baddomain/',
   'http://slashdot.org/',
  'http://reddit.com/',
  'http://news.ycombinator.com/'
 ]


def dwld(url):
  print url
  res = urllib.urlopen(url).read() 
  filename = tldextract.extract(url)
  with open(filename.domain, 'wb') as fh:
     fh.write(res)
  return url 

pool = ThreadPool(processes = 4)
pool.map(dwld, URLs)

Gives me Traceback (most recent call last): File "dwld_thread.py", line 26, in pool.map(dwld, URLs) File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py", line 148, in map return self.map_async(func, iterable, chunksize).get() File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py", line 422, in get raise self._value IOError: [Errno socket error] [Errno 8] nodename nor servname provided, or not known

Improve this question
2
  • What is that "stream" library you're using? Commented Jan 22, 2014 at 7:38
  • 1
    I think the problem with your second version is that you want to call pool.close() and pool.join() (or, better, just use a with statement, if that works in your version… I forget when multiprocessing Pools became context managers). Commented Jan 22, 2014 at 8:03

4 Answers 4

Reset to default
3

I have no idea what that stream.ThreadPool is that you're using, or what its API is… but the problem is obvious:

res = urllib.urlretrieve(url).read()

If you look at the doc for urlretrieve:

Return a tuple (filename, headers) where filename is the local file name under which the object can be found…

You obviously can't call read on that. If you want to download to a local file, using this legacy API, and then read that file, you can:

filename, headers = urllib.urlretrieve(url)
with open(filename) as f:
    res = f.read()

But why? Just use urllib2.urlopen, which "returns a file-like object with two additional methods", so you can just call read on it, and you won't be creating a temporary file, and you're not using an old function that wasn't quite designed right that nobody has maintained in years.

But Python has a nice ThreadPoolExecutor built into the standard library. And if you look at the very first example they show you, it's exactly what you're trying to do.

Unfortunately, you're using Python 2.x, which doesn't have the concurrent.futures module. Fortunately, there is a backport on PyPI that works with 2.5+.

Python also has multiprocessing.dummy.Pool (also available under the undocumented, but probably more readable, name multiprocessing.ThreadPool). But if you're willing to go outside the stdlib for some module that you apparently aren't sure how to use and that I've never heard of, I'm guessing you won't have any problem using futures. So:

import futures
import urllib2

URLs = [
  'http://www.cnn.com/',
  'http://www.bbc.co.uk/',
  'http://www.economist.com/',
  'http://nonexistant.website.at.baddomain/',
  'http://slashdot.org/',
  'http://reddit.com/',
  'http://news.ycombinator.com/'
 ]

def load_url(url):
    return urllib2.urlopen(url).read()

if __name__ == '__main__':
    with futures.ThreadPoolExecutor(max_workers=7) as executor:
        fmap = dict((executor.submit(load_url, url), url) for url in URLs)
        for f in futures.as_completed(fmap):
            url = fmap[f]
            try:
                content = f.result()
            except Exception as exception:
                print '%r failed: %s' % (url, exception)
            else:
                print '%r is %d bytes' % (url, len(content))
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Throws an error at this line: fmap = {executor.submit(load_url, url): url for url in URLS}
fmap = {executor.submit(load_url, url): url for url in URLs} ^ SyntaxError: invalid syntax
@wannaC: Ah, Python 2.6 I'll bet, right? So you don't have dict comprehensions. Edited (but not tested; please let me know). Really, the whole dict of futures: URLs thing is overly complicated; I just did it that way to mirror the example in the docs…
Yep it's 2.6 Code works now. ALso it asked me to use concurrent.futures instead of futures. Just one question if I want to check whether the link is pdf of or not where should I check it?
@wannaC: You want to check based on the extension on the URL, by looking at the file magic of the returned content, by trying to open it with a PDF library, or what? You can do any of those either on the background thread as soon as you read the data, or on the main thread where you're doing len(content). The only constraint is that if it takes significant CPU work, don't do it in the background threads.
|
0

urllib.urlretrieve(url).read() should be urllib.urlopen(url).read()

Improve this answer

Comments

0 
from threading import *
from time import sleep
# if Python2:
import urllib
# if Python3:
# import urllib.request

URLs = [
  'http://www.cnn.com/',
  'http://www.bbc.co.uk/',
  'http://www.economist.com/',
  'http://nonexistant.website.at.baddomain/',
  'http://slashdot.org/',
  'http://reddit.com/',
  'http://news.ycombinator.com/'
 ]

class worker(Thread):
    def __init__(self, link):
        Thread.__init__(self)
        self.link = link
        self.start()
    def run(self):
        # if Python2:
        res = urllib.urlopen(url).read() # as mentioned by @DhruvPathak
        # if Python3:
        # res = urllib.request.urlopen(url).read()
        with open(url, 'rb') as fh:
            fh.write(res) # store fetched data in a file called <link>

for url in urls:
    while len(enumerate()) > 500:
        sleep(0.25)
    worker(url)

while len(enumerate()) > 1:
    sleep(0.25) # wait for all threads to finish
Improve this answer

12 Comments

Shouldn't open contain wb instead of rb? Also it downloads just the last url not all the urls.
Also, if you wanted to expand this to 5000 URLs, you wouldn't want to spawn 5000 threads. The OP is trying to use a thread pool, which would solve that problem; why make his code worse?
Also, if you want to wait for all threads to finish, spinning on enumerating the threads over and over again is a very bad way to do that, unless you're trying to make sure you burn 100% CPU and cause horrible GIL conflicts. That's what join is for. Just do workers = [worker(url) for url in urls], then for worker in workers: worker.join().
And it tries to save the files with filenames like http://www.cnn.com/, which isn't going to work too well on most platforms. (You'll probably just get an error about no such relative path as http:.)
@abarnert Haters gona hate, and yes this code wasn't complete. I rarely go on to Stack Overflow to do all the job for people, some things should be left alone and for the user to figure out. But indeed there should be a thread-pool since there's many links, so i added one. and enumerate() isn't to taxing and can be used for this purpose. And yes, the filenames would be bad and who in their right mind would call them http:/google.com/, i know i wouldn't.. but the user has his own preference so i'll leave for the user to descide his naming convention since it's not a part of the OP question.
|
0

What about using multiprocessing ?

Sample code:

#! /usr/bin/env python

# -*- coding: utf-8 -*-


import sys
import urllib
from multiprocessing import Pool

import os

POOL = 8
PDFS_DOWNLOAD_DIR = 'pdfs'
PDF_LINKS = sys.argv[1]


class DownloadFiles(object):
    def __init__(self):
        self.pdf_links = self.read_links_from_file()
        self.create_download_dir()

    def create_download_dir(self):
        try:
            if not os.path.exists(PDFS_DOWNLOAD_DIR):
                os.makedirs(PDFS_DOWNLOAD_DIR)
        except IOError as e:
            exit()

    def read_links_from_file(self):
        try:
            with open(PDF_LINKS, 'r') as f:
                return list(set([x.strip() for x in f]))
        except (IndexError, IOError) as e:
            exit()

    def get_file(self, link):

        filename = link.split('/')[-2]

        print('Downloading file --> "{filename}"'.format(
            filename=filename
        ))

        urllib.urlretrieve(link, filename='{pdfs_data}/{filename}'.format(
            pdfs_data=PDFS_DOWNLOAD_DIR,
            filename=filename
        ))

    def download(self):

        pool = Pool(POOL)
        pool.map(self.get_file, self.pdf_links)

        pool.close()
        pool.join()

        print('\nSuccessfully downloaded files from given source!\n')


d = DownloadFiles()
d.download()
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.