Max number of Python scripts running simultaneously

Question

I have written a script to do some research on HTTP Archive data. This script needs to make HTTP requests to sites scraped by HTTP Archive in order to classify sites into groups (e.g., Drupal, WordPress, etc). The script is working really well; however, the list of sites that I am handling is 300,000 sites long.

I would like to be able to complete the categorization of sites as fast as possible. I have experimented with running multiple instances of the script at the same time and it is working well with appropriate locks in place to prevent race conditions.

How can I max this out to get all of these operations completed as fast as possible? For instance, I am looking at spinning up a VPS with 8 CPUs and 16 GB RAM. How do I maximize these resources to make sure I'm using every bit of processing power possible? I may consider spinning up something more powerful, but I want to make sure I understand how to get the most out of it so I'm not wasting money.

Thanks!

I'm guessing more io bound than CPU bound; although, networking is likely going to be the real bottleneck. — tollmanz
– tollmanz, Commented Sep 7, 2014 at 13:28

Masih · Accepted Answer · 2014-09-07 06:25:05Z

1

Multiprocessing module is the best option that lets you harness the maximum power of your 8 CPUs: https://docs.python.org/3.3/library/multiprocessing.html

answered Sep 7, 2014 at 6:25

Masih

9902 gold badges21 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Max number of Python scripts running simultaneously

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related