How to insert multiple arguments in multiprocessing

Question

I have this project I'm working on and I'm trying to implement multiprocessing. I can't quite get it to work because the function I'm trying to have it run requires multiple arguments. Basically, there are sites that processDFStandardCurve processes and that is the only variable that changes in each test that multiprocess would be running. Those different site names are stored as siteIDList.

def multiprocess (cursor, testsDict, optionsDict, outputPath, calculated_pdf, stationToPriority, result):
    siteIDList = []
    for line in result:
        siteID = line[3]
        nbsNum = line[2]
        if siteID != "":
            siteIDList.append(siteID)
        else:
            siteIDList.append(nbsNum)
    p = multiprocessing.Pool()
    p.starmap(processDFStandardCurve, siteIDList,  cursor, testsDict, optionsDict, outputPath, calculated_pdf, stationToPriority)

processDFStandardCurve is as follows:

def processDFStandardCurve(cursor, siteID, testsDict, optionsDict, outputPath, calculated_pdf, stationToPriority):

It all stops after the p.map() call.

When using starmap it throws this error

TypeError: starmap() takes from 3 to 4 positional arguments but 9 were given

Nathaniel Ford · Accepted Answer · 2022-08-22 17:33:44Z

To invoke starmap you want to do something like this:

p.starmap(processDFStandardCurve, [(siteIDList,  cursor, testsDict, optionsDict, outputPath, calculated_pdf, stationToPriority)])

The arguments must be in their own iterable. We can see this in the documentation (linked above) of the signature:

starmap(func, iterable[, chunksize])

This means it takes two required and one optional. The iterable provided here are a list of all the arguments that are unpacked and sent to func. Each item in the outer iterable is passed to the function provided in the first value. Therefore the inner iterable should be your arguments.

In your case you need to modify your code somewhat:

def multiprocess (cursor, testsDict, optionsDict, outputPath, calculated_pdf, stationToPriority, result):
    arg_list = []
    for line in result:
        id = line[3] if line[3] != "" else line[2]
        site_args = (id, cursor, testsDict, optionsDict, outputPath, calculated_pdf, stationToPriority)
        arg_list.append(site_args)

    p = multiprocessing.Pool()
    p.starmap(processDFStandardCurve, arg_list)

Daniel Fainberg · Accepted Answer · 2022-08-22 16:58:36Z

0

Check out starmap. You can pass multiple arguments to it.

answered Aug 22, 2022 at 16:58

Daniel Fainberg

20.1k3 gold badges43 silver badges80 bronze badges

5 Comments

EthanMcQ Over a year ago

I've tried starmap, but it says that I can use 3 to 4 arguments but there are 9 or so. I'm not sure if there is a solution for that?

Charles Duffy Over a year ago

"three to four"? starmap has no limit on number of arguments. Can you show us how you tried and how it failed?

Charles Duffy Over a year ago

@EthanMcQ, ...I suspect that maybe you passed the arguments directly on starmap's parameter list, instead of making them list entries inside the iterable parameter. There shouldn't be any hardcoded limit to how many items can be inside iterable, or how many parameters each of those items can have.

EthanMcQ Over a year ago

I just edited the post to show the error with starmap

Nathaniel Ford Over a year ago

Starmap does have a constrained number of arguments: self, a function, an iterator of iterators of arguments, and an optional chunk size. That's where the 3-4 comes from.

Collectives™ on Stack Overflow

How to insert multiple arguments in multiprocessing

2 Answers 2

Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related