11

We have a about 500GB of images in various directories we need to process. Each image is about 4MB in size and we have a python script to process each image one at a time (it reads metadata and stores it in a database). Each directory can take 1-4 hours to process depending on size.

We have at our disposal a 2.2Ghz quad core processor and 16GB of RAM on a GNU/Linux OS. The current script is utilizing only one processor. What's the best way to take advantage of the other cores and RAM to process images faster? Will starting multiple Python processes to run the script take advantage of the other cores?

Another option is to use something like Gearman or Beanstalk to farm out the work to other machines. I've taken a look at the multiprocessing library but not sure how I can utilize it.

Improve this question
4
  • possible duplicate of How to use all the cores in Windows 7? Commented Apr 4, 2012 at 14:22
  • 1
    If you already have a working script that you can point at whichever files/directories you want, consider writing a shell script to spin up as many instances as you need. Commented Apr 4, 2012 at 14:23
  • 4
    I'd start by looking at where the bottle necks are. If you are spending the bulk of your time performing IO then you aren't going to be able to go much faster. For example, if a single process manages to load data from the disk at a certain speed, it's highly unlikely that you'll be able to maintain that speed when multiple processes are chewing the disk. Commented Apr 4, 2012 at 14:24
  • I really agree with Jeff Forster, look first for bottle necks. If the only treatment you do on files is reading meta data and store them, there is an high chance that your bottle neck is disk access. Therefore, trying to add more process accessing the same disk will only degrade the performance. Commented Apr 4, 2012 at 15:10

6 Answers 6

Reset to default
6

Will starting multiple Python processes to run the script take advantage of the other cores?

Yes, it will, if the task is CPU-bound. This is probably the easiest option. However, don't spawn a single process per file or per directory; consider using a tool such as parallel(1) and let it spawn something like two processes per core.

Another option is to use something like Gearman or Beanstalk to farm out the work to other machines.

That might work. Also, have a look at the Python binding for ZeroMQ, it makes distributed processing pretty easy.

I've taken a look at the multiprocessing library but not sure how I can utilize it.

Define a function, say process, that reads the images in a single directory, connects to the database and stores the metadata. Let it return a boolean indicating success or failure. Let directories be the list of directories to process. Then

import multiprocessing
pool = multiprocessing.Pool(multiprocessing.cpu_count())
success = all(pool.imap_unordered(process, directories))

will process all the directories in parallel. You can also do the parallelism at the file-level if you want; that needs just a bit more tinkering.

Note that this will stop at the first failure; making it fault-tolerant takes a bit more work.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

4

Starting independent Python processes is ideal. There will be no lock contentions between the processes, and the OS will schedule them to run concurrently.

You may want to experiment to see what the ideal number of instances is - it may be more or less than the number of cores. There will be contention for the disk and cache memory, but on the other hand you may get one process to run while another is waiting for I/O.

Improve this answer

Comments

4

You can use pool of multiprocessing to create processes for increasing performance. Let's say, you have a function handle_file which is for processing image. If you use iteration, it can only use at most 100% of one your core. To utilize multiple cores, Pool multiprocessing creates subprocesses for you, and it distributes your task to them. Here is an example:

import os
import multiprocessing

def handle_file(path):
    print 'Do something to handle file ...', path

def run_multiprocess():
    tasks = []

    for filename in os.listdir('.'):
        tasks.append(filename)
        print 'Create task', filename

    pool = multiprocessing.Pool(8)
    result = all(list(pool.imap_unordered(handle_file, tasks)))
    print 'Finished, result=', result

def run_one_process():
    for filename in os.listdir('.'):
        handle_file(filename)

if __name__ == '__main__':
    run_one_process
    run_multiprocess()

The run_one_process is single core way to process data, simple, but slow. On the other hand, run_multiprocess creates 8 worker processes, and distribute tasks to them. It would be about 8 times faster if you have 8 cores. I suggest you set the worker number to double of your cores or exactly the number of your cores. You can try it and see which configuration is faster.

For advanced distributed computing, you can use ZeroMQ as larsmans mentioned. It's hard to understand at first. But once you understand it, you can design a very efficient distributed system to process your data. In your case, I think one REQ with multiple REP would be good enough.

enter image description here

Hope this would be helpful.

Improve this answer

Comments

2

See the answer to this question.

If the app can process ranges of input data, then you can launch 4 instances of the app with different ranges of input data to process and the combine the results after they are all done.

Even though that question looks to be Windows specific, it applies to single threaded programs on all operating system.

WARNING: Beware that this process will be I/O bound and too much concurrent access to your hard drive will actually cause the processes as a group to execute slower than sequential processing because of contention for the I/O resource.

Improve this answer

Comments

0

If you are reading a large number of files and saving metadata to a database you program does not need more cores.

Your process is likely IO bound not CPU bound. Using twisted with proper defereds and callbacks would likely outperform any solution that sought to enlist 4 cores.

Improve this answer

Comments

0

I think in this scenario it would make perfectly sense to use Celery.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.