Correctly implementing Python Multiprocessing

Question

I was going through the multiprocessing tutorial here: http://pymotw.com/2/multiprocessing/basics.html

I wrote the script below as an exercise. The script appears to be working and I do see 5 new python processes running in taskmgr. However, my print statement outputs the same folder being searched multiple times.

I have a suspicion that instead of splitting up the work between different processes, I'm giving each process the entire work load. I'm pretty certain I'm doing something wrong and inefficiently. Could someone please point out my errors?

What I have so far:

def email_finder(msg_id):
    for folder in os.listdir(sample_path):
        print "Searching through folder: ", folder
        folder_path = sample_path + '\\' + folder
        for file in os.listdir(os.listdir(folder_path)):
            if file.endswith('.eml'):
                file_path = folder_path + '\\' + file
                email_obj = email.message_from_file(open(file_path))
                if msg_id in email_obj.as_string().lower()
                    shutil.copy(file_path, tmp_path + '\\' + file)
                    return 'Found: ', file_path
    else:
        return 'Not Found!'

def worker():
    msg_ids = cur.execute("select msg_id from my_table").fetchall()
    for i in msg_ids:
        msg_id = i[0].encode('ascii')
        if msg_id != '':
            email_finder(msg_id)
    return

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        p = multiprocessing.Process(target=worker)
        jobs.append(p)
        p.start()

shx2 · Accepted Answer · 2013-03-10 16:54:44Z

5

Each of your subprocesses gets its own cursor, and therefor iterates over the entire set of IDs.

You need to read the msg_ids from your DB once, and then spawn it to sub-processes, instead of letting each sub-process query on its own.

answered Mar 10, 2013 at 16:54

shx2

64.8k17 gold badges139 silver badges166 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Dirty Penguin Over a year ago

Done :) The UI forced me to wait for a minute before accepting.

Collectives™ on Stack Overflow

Correctly implementing Python Multiprocessing

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related