I was wondering how I could limit something like this too use only 10 threads at one time
with open("data.txt") as f:
for line in f:
lines = line.rstrip("\n\r")
t1 = Thread(target=Checker, args=("company"))
t1.start()
-
Just monitor the threads alive and block until one of the threads exits if there are 10 of them or more.ForceBru– ForceBru2017-01-21 15:28:01 +00:00Commented Jan 21, 2017 at 15:28
-
Possible duplicate of How do I limit the number of active threads in python?zondo– zondo2017-01-21 15:30:36 +00:00Commented Jan 21, 2017 at 15:30
-
Possible duplicate of Python thread pool similar to the multiprocessing Pool?Amit Kumar– Amit Kumar2017-01-21 15:32:05 +00:00Commented Jan 21, 2017 at 15:32
-
Does this answer your question? How to limit number of concurrent threads in Python?feetwet– feetwet2023-08-29 01:36:24 +00:00Commented Aug 29, 2023 at 1:36
Add a comment
|
4 Answers
Use Python's ThreadPoolExecutor with max_workers argument set to 10.
Something like this:`
pool = ThreadPoolExecutor(max_workers=10)
with open("data.txt") as f:
for line in f:
lines = line.rstrip("\n\r")
pool.submit(Checker,"company")
pool.shutdown(wait=True)
The pool will automatically allocate threads as needed, limiting maximum number of allocation to 10. The first argument in pool.submit() is the function name, the arguments are simply passed as comma-separated values.
pool.shutdown(wait=True) waits for all threads to complete execution.
answered Jan 21, 2017 at 15:33
Shihab Shahriar Khan
5,5352 gold badges20 silver badges28 bronze badges
Comments
Use the ThreadPoolExecutor and tell it that you want 10 threads.
def your_function_processing_one_line(line):
pass # your computations
with concurrent.futures.ThreadPoolExecutor(10) as executor:
result = executor.map(your_function_processing_one_line, [line for line in f])
...and you will have all the results in result.
2 Comments
Raheel
what if multiple params ?
yogabonito
This is also possible. Take a look at this nice answer.
I wrote this nested loop to cap threads to a variable. This code relies on a preset array of commands to process. I have borrowed some elements from other answers for thread launch.
import os, sys, datetime, logging, thread, threading, time
from random import randint
# set number of threads
threadcount = 20
# alltests is an array of test data
numbertests = len(alltests)
testcounter = numbertests
# run tests
for test in alltests:
# launch worker thread
def worker():
"""thread worker function"""
os.system(command)
return
threads = []
t = threading.Thread(target=worker)
threads.append(t)
t.start()
testcounter -= 1
# cap the threads if over limit
while threading.active_count() >= threadcount:
threads = threading.active_count()
string = "Excessive threads, pausing 5 secs - " + str(threads)
print (string)
logging.info(string)
time.sleep(5)
# monitor for threads winding down
while threading.active_count() != 1:
threads = threading.active_count()
string = "Active threads running - " + str(threads)
print (string)
logging.info(string)
time.sleep(5)
Comments
(for both Python 2.6+ and Python 3)
Use the threadPool from multiprocessing module:
from multiprocessing.pool import ThreadPool
The only thing is that it is not well documented...
