Is there a maximum limit on running concurrent threads (Python)?

Question

I am reading hundreds of tweets where I inspect the URLs. I am using multithreading for this task as the URL reading takes more than a second. However, I am not sure how many threads can I run in parallel at a time in this situation?

import Queue
import threading
q = Queue.Queue()
thread limit = 100

for tweet in tweets[0:threadlimit]:
            t = threading.Thread(target=process_tweet, args=(q, tweet))
            t.daemon = True
            t.start()


    for tweet in tweets[0:threadlimit]:
          tweet = q.get()

The reason I am asking this that when I use a thread limit of 100 then it works fine but for a threadlimit of 200, it gets stuck.

Platform: Linux

Basile Starynkevitch · Accepted Answer · 2017-08-21 12:14:12Z

The operating system is always having some limits on the number of threads, and each thread uses some resources (notably some space, perhaps a megabyte, for the thread's call stack). So it is not reasonable to have lots of threads. Details are operating system and computer specific. On Linux, see getrlimit(2) for RLIMIT_STACK (the default stack size) and RLIMIT_NPROC (number of processes, actually tasks, including threads, you are permitted to have).. and also pthread_attr_setstacksize(3) & pthread_create(3).

Threads are often heavy on resources (so read about green threads). You don't want to have many (e.g. thousands, or even a hundred) of them on a laptop or desktop (some supercomputers or costly servers have hundreds of cores with NUMA, then you could try having more threads).

Collectives™ on Stack Overflow

Is there a maximum limit on running concurrent threads (Python)?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related