What I need - to have 20 active threads. No more. I need the way to control that the number is no more than 20. And in the end of python script, i need to make sure if all this threads are closed. How to do so?
First try - to sleep 0.1 second. But it is pretty random decision and I have error from time to time: Connection pool is full, discarding connection
list_with_file_paths = [...]
slice_20 = list_with_file_paths[:20]
threads = []
for i in slice_20:
th = threading.Thread(target=process_data, args=(i,))
print('active count:', threading.active_count())
if threading.active_count() < 20:
th.start()
else:
time.sleep(0.1)
th.start()
threads.append(th)
Second try - to append all threads in one list and .join() them. But it is not connected with number of 20 threads.
list_with_file_paths = [...]
slice_20 = list_with_file_paths[:20]
threads = []
for j in range(10): # I take every 20 items from list_with_file_paths but I have shortened for this example
for i in slice_20:
th = threading.Thread(target=process_data, args=(i,))
print('active count:', threading.active_count())
if threading.active_count() < 20:
th.start()
else:
time.sleep(0.1)
th.start()
threads.append(th)
for thread in threads:
thread.join()
Is Limit number of active threads Python only chance to limit active threads number?
ThreadPool