Easiest (least amount of boilerplate code) way to parallelize a python loop?

Question

I have some code that looks like this:

for photo in photoInfo:
    if not('url' in photo):
        raise Exception("Missing URL: " + str(photo) + " in " + str(photoInfo))
    sizes = getImageSizes(photo['url'])
    photo.update(sizes)

It might not be obvious, but the code performs a mix of high-latency I/O (opening a remote URL) and moderately CPU-intensive process (parsing image and extracting size) for each photo.

What's the easiest way to parallelize this code?

What I have tried so far

I found this code in the answer to another, more complex question, but I'm having a hard time mapping it back to my much simpler use-case:

from itertools import product
from multiprocessing import Pool

with Pool(processes=4) as pool:  # assuming Python 3
    pool.starmap(print, product(range(2), range(3), range(4)))

Abdul Majeed · Accepted Answer · 2019-05-24 23:52:46Z

0

from multiprocessing import Pool
import os

def user_defined_function(url):
    #your logic for a single url
    pass

if __name__ == '__main__':
    urls_list = ['u1','u2']
    pool = Pool(os.cpu_count())                         # Create a multiprocessing pool
    pool.map(user_defined_function, urls_list)

it's sample code you can modify it according to your usage. I will map each element of the list to your function and perform it individually.

answered May 24, 2019 at 23:52

Abdul Majeed

2,80127 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

blhsing · Accepted Answer · 2019-05-25 00:10:34Z

0

You can use Pool.map to parallelize the fetching of image sizes, and build a new dict with the returning values and the same keys:

from multiprocessing import Pool

def get_image_size(photo):
    if 'url' not in photo:
        raise Exception("Missing URL: " + str(photo))
    return getImageSizes(photo['url'])

if __name__ == '__main__':
    with Pool() as pool:
        photoInfo = dict(zip(photoInfo, pool.map(get_image_size, photoInfo)))

edited May 25, 2019 at 0:10

answered May 24, 2019 at 23:45

blhsing

109k9 gold badges88 silver badges132 bronze badges

Collectives™ on Stack Overflow

Easiest (least amount of boilerplate code) way to parallelize a python loop?

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related