5

I have recently discovered how awesome Pool.map() is for simple multiprocessing tasks, but my local machine only has two processors and I was hoping to take advantage of some remote computing resources.

Can a processing pool be constructed to use remote machines as well as the local machines in a transparent way?

What python code would I need to run on the remote machines, How would I set up the pool on the local (master) machine?

On the documentation page for multiprocessing, It says Managers are to be used for shared objects, but it's unclear to me how they should be used for a remote Pool. Nothing other than Managers seems make any reference to ports and addresses, so I'm guessing all remote activity happens through this class.

Improve this question
2
  • 2
    Perhaps this is helpful: stackoverflow.com/questions/5181949/… Commented Apr 11, 2011 at 13:04
  • Thanks's that was very helpful. I think jug (a link from that page) is just what I needed. Commented Apr 12, 2011 at 19:49

1 Answer 1

Reset to default
2

multiprocessing.Pool seems not designed for distributed processing -- if that's what you want to do, I suggest you check out the execnet package instead. That works over SSH pipes and sends code along to the remote Python interpreter if necessary.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.