How to increase numbers of cpu cores and their usage in python

Question

I've written a python script with heavy computational costs. After running the code in a remote linux server(via putty), it takes one week to get finished.

Our remote server has 32 cpu cores and 64 GB of Ram memory. I need to run my script in a shorter time by using more numbers of cpu cores but I don't know exactly how to increase using more numbers of cpu cores for my script.

I would like to know if there is any python code to add at the first or end part of my python script for adding numbers of cpu cores, memory usage and etc to be able to run the script in a shorter time. I also need to return to the default state after getting my script result.

Is there any way to achieve this in Python?

Have a look at pythons multiprocessing library

Tom
– Tom

2018-01-31 11:05:32 +00:00
Commented Jan 31, 2018 at 11:05 — Tom
– Tom, Commented Jan 31, 2018 at 11:05

srjjio · Accepted Answer · 2018-01-31 11:15:10Z

2

You can take a look on multiprocessing lib (https://docs.python.org/2/library/multiprocessing.html) but you will have to manage data to share between processes.

You can also take a look on threading lib (https://docs.python.org/2/library/threading.html) which may be efficient if you have lot of I/O.

Finally you use asyncio lib (python 3.4+ https://docs.python.org/3/library/asyncio.html) which is, in my opninion, more complicated to implement but give you more ability to manage explicitly the behavior of your code.

Edit: This is a large question, you may find lot of documentation on this issue. It really depends on what your code really do, you should better give a snippet of your code.

edited Jan 31, 2018 at 11:15

answered Jan 31, 2018 at 11:08

srjjio

1,0491 gold badge9 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Thiago F. Pappacena · Accepted Answer · 2018-01-31 11:06:30Z

1

Rewrite your script to do things in parallel. Since threads don't use more than one CPU in CPython, you should use https://docs.python.org/2/library/multiprocessing.html.

See also: Does python support multiprocessor/multicore programming?

answered Jan 31, 2018 at 11:06

Thiago F. Pappacena

3261 silver badge5 bronze badges

Collectives™ on Stack Overflow

How to increase numbers of cpu cores and their usage in python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related