0

I have written a python code to carry out genetic algorithm optimization, but it is too slow. I would like to know how to run the same in parallel mode making use of multiple CPUs ?

For more clarity, another python code will be called by my code for, say 100 times one after the other, I wanted to divide this between 4 CPUs. So that 25 times the outside python code is solved by each CPU. Thereby increasing the speed. Its highly appreciated if someone can help me with is ?

Thanks in advance!

Improve this question
0

1 Answer 1

Reset to default
2

There are several packages that provide parallel computing for python2. I am the author of a package called pathos, which provides parallel computing with several parallel backends and gives them a common API. pathos provides parallel pipes and maps for multi-process, multi-threaded, parallel over sockets, MPI-parallel, and also interactions with schedulers and over ssh. pathos relies on several packages, which you can pick from if you don't want all the different options.

pathos uses: pyina which in turn uses mpi4py. mpi4py provides bindings to MPI, but you can't run the code from python 'normally'… you need to run with whatever you use to run MPI. pyina enables you to run mpi4py from normal python, and to interact with schedulers. Plus, pyina uses dill, which can serialize most python objects, and thus you are much more able to send what you want across processes.

pathos provides a fork of multiprocessing that also plays well with dill and pyina. Using both can enable you to do hierarchical parallel competing -- like launching MPI parallel that then spawns multiprocess or multithreaded parallel.

pathos also uses ppft, which is a fork of pp (Parallel Python), which provides parallel computing across sockets -- so that means you can connect a parallel map across several machines.

There are alternatives to pathos, such as IPython-parallel. However, the ability to use MPI is very new, and I don't know how capable it is yet. It may or may not leverage IPython-cluster-helper, which has been in development for a little while. Note that IPython doesn't use pp, it uses zmq instead, and IPython also provides connectivity to EC2 if you like cloud stuff.

Here are some relevant links:

  1. pathos, 'pyina, dill, ppft: https://github.com/uqfoundation
  2. Ipython-cluster-helper: https://github.com/roryk/ipython-cluster-helper
  3. IPython: http://ipython.org/ipython-doc/dev/parallel/parallel_mpi.html
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Note that the latest stable release of pathos and pyina are fairly stale, and a new releases are due out imminently. At the moment, the best approach is to use the versions on github.
Also, if you'd like a fast GA algorithm that already leverages pathos, pyina, and pptf, then you might want to try mystic. It's an optimization framework that's built to run on parallel/distributed computing. It also has parallel ensemble solvers, which can provide global optimization at orders of magnitude faster than a GA -- if you have enough parallel resources. mystic is also found at github.com/uqfoundation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.