How to run multiprocessing python request with function with multiple agruments

Question

I'm trying to gather data from a single url using python requests library.

I want to run multiprocessing to speed up the data collection however I get an error when I pass in the argument of my function inside of Pool.

Kindly note I have already read the following previous questions:

a link and a link however none of those answer my question.

How can I run those get request simultaneously passing the 3 mandatory arguments?

Here is my code:

from multiprocessing import Pool
import requests
url = 'http://icanhazip.com'
url_two = 'https://httpbin.org/ip'
url_three = 'https://httpbin.org/get'
start_point = 'a'
start_point_two = 'b'
start_point_three ='c'
ending_point = 'c'
ending_point_two = 'z'
ending_point_three = 'x'


def get_info(url,start_point,ending_point):
    r = requests.get(url)
    html = r.text
    if start_point in html:
        print('Do Something')
    elif ending_point in html:
        print('Do Something else')
   else:
        pass

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(get_info, [[url,start_point,ending_point]]))

This is the error I get:

TypeError: get_info() missing 2 required positional arguments: 'start_point' and 'ending_point'

use one argument get_info(args) and upack it inside get_info - url,start_point,ending_point = args — furas
– furas, Commented Jul 21, 2019 at 17:40

RomanPerekhrest · Accepted Answer · 2019-07-22 07:22:19Z

1

To pass multiple arguments to a target function - use Pool.starmap feature:

In your case it'd look as below:

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.starmap(get_info, [(url, start_point, ending_point),
                               (url_two, start_point_two, ending_point_two),
                               (url_three, start_point_three, ending_point_three),]

edited Jul 22, 2019 at 7:22

answered Jul 21, 2019 at 18:20

RomanPerekhrest

93.1k4 gold badges75 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Pro Girl Over a year ago

Hi @RomanPerekhrest, it worked with the .starmap suggestion you gave, can you please modify the answer replacing ("map" with "starmap" after print) so that I can assign you the answer and people reading it in future will have the solution ready and correct? Thanks

RomanPerekhrest Over a year ago

@SkylerX, sorry, sure. That was a typo, fixed now

Collectives™ on Stack Overflow

How to run multiprocessing python request with function with multiple agruments

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related