Batching threads in python

Question

I'm updating some existing code that looks like this:

for i in list
  thread.start_new_thread(main, (i[0],i[1],i[2],2))

This causes threads to be created for every item in the list, but not executed until all the threads are created. I'd like to either execute the threads in small groups, or just have them execute directly after they're created.

(There are a lot of python threads discussion on here so sorry if I missed this type of question already being asked...)

start_new_thread as its name suggests starts new thread immediately. Note thread module had been deprecated by threading module since 2005. — jfs
– jfs, Commented Dec 2, 2012 at 20:51

lqc · Accepted Answer · 2012-12-02 19:17:26Z

4

You may find the concurrent.futures module useful. It's available also for Python 2 under the name futures. For example, to invoke a function main for every item on MY_LIST concurrently, but with at most 5 threads, you can write:

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor(max_workers=5) as executor:
    for result in executor.map(main, MY_LIST):
        # Do something with result
        print(result)

edited Dec 2, 2012 at 19:17

answered Dec 2, 2012 at 15:17

lqc

7,4081 gold badge28 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Lizbeth Over a year ago

This worked pretty well actually. I feel I should be able to implement this in native code. No?

Collectives™ on Stack Overflow

Batching threads in python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related