threading, multiprocessing error

Question

It's the same as finder on Windows but use thread to get faster,

    import os,threading,multiprocessing


    def finder(path,q):
     for x in os.walk(unicode(path)):
      if x[1]:
       for dirname in x[1]:
        if target in dirname.lower():
         q.put(os.path.join(x[0],dirname))
      if x[2]:
       for name in x[2]:
        if target in name.lower():
         q.put(os.path.join(x[0],name))

     q.put(1)

    def printer(q):
     cmd=0
     while 1:
      tmp=q.get()
      if tmp==1:
       cmd += 1
       continue
      if cmd ==thnum:
       break
      print tmp

    if __name__ =="__main__":
     q=multiprocessing.JoinableQueue()
     ini=os.walk(u"C:\\").next()
     thnum=len(ini[1])
     target=raw_input("what you wanna get\n")

     p=multiprocessing.Process(target=printer,args=(q,))
     p.daemon=1
     p.start()

     for i in xrange(thnum):
      t=threading.Thread(target=finder,args=(ini[1][i],q,))
      t.start()
      print i," started"
     q.join()

it shows

0 started 1 started .... 22 started

but never shows the result so my question is

why doesn't the result shows
I know the code is dirty:(...is that a clean way to do it?

thank you guys.

Check out the Python Style Guide for examples of how to write "clean" code. — Joel Cornett
– Joel Cornett, Commented Jul 9, 2012 at 1:35
Also, what makes you think that multithreading will be faster? — Joel Cornett
– Joel Cornett, Commented Jul 9, 2012 at 1:36
@JoelCornett thankyou! but what I wondered is about the algorithm or more appropriate function:) — from __future__
– from __future__, Commented Jul 9, 2012 at 1:41
This code seems to be IO bound, so I doubt multiprocessing is any faster. The GIL really isn't a factor I think — jdi
– jdi, Commented Jul 9, 2012 at 1:47

jdi · Accepted Answer · 2012-07-09 03:20:27Z

You have just a ton of messy code in here and also some errors. The major problem I see is that your threads are immediately failing to produce anything from their os.walk, and going right to exiting with the q.put. This is because you don't pass a full path to each thread. Only a directory name. But its hard to know this because you dont use descriptive names for any variables.

Here is a cleaned up version:

import os
import threading
import multiprocessing


def finder(path, q, done):
    for root, dirs, files in os.walk(unicode(path)):
        for dirname in dirs:
            if target in dirname.lower():
                q.put(os.path.join(root,dirname))
        for name in files:
            if target in name.lower():
                q.put(os.path.join(root,name))

    # print "Leaving thread", threading.current_thread()
    done.put(1)

def printer(q,done,worker_count):
    total = 0
    while 1:
        try: done.get_nowait()
        except: pass
        else: total += 1

        if total == worker_count:
            break

        try: tmp=q.get(timeout=1)
        except: pass

        print tmp

if __name__ =="__main__":

    results = multiprocessing.Queue()
    done = multiprocessing.Queue()
    root, dirs, files = os.walk(u"C:\\").next()
    thnum=len(dirs)
    target=raw_input("what you wanna get\n")

    p=multiprocessing.Process(target=printer,args=(results,done,thnum))
    p.start()

    for i in xrange(thnum):
        full_path = os.path.join(root, dirs[i])
        t=threading.Thread(target=finder,args=(full_path, results, done))
        t.start()

    p.join()

See how I join the full path together in the main block before sending them off to each thread? I removed the JoinableQueue because it was never going to do what you think it was. If at any time the printer has cleared out the results queue, but the threads are still trying to find more, the queue will think its done and exit. What I replaced it with is another queue to be used as a signal. Each worker puts an item in the queue when its done. Then the printer keeps checking to see if it can pull enough signals from the done queue to equal the amount of workers launched. If so, it will exit.

This whole thing could still be rewritten better, but I am just applying bandaids to what you have. I sort of just threw this together with what you had.

Note, the way you start the whole process, checking for the directories under the starting path, will basically just exit out if there are only files.

You are the Savior!! I realize not only the problems, but the significance of clean naming. Thank you very much
Not a problem my friend. Ya, it totally helps us SO people out when we have to review the code. Far easier to see the intent of each line when you see a descriptive name.

pinkdawn · Accepted Answer · 2012-07-09 02:05:53Z

1

for your second, to write a clean multi-thread code, using decorators helps you, and make it eaiser switch between thread and process.

check the example here decorators async decorator

You could install the decorators by:

 easy_install decorator

or download the code, using python setup.py install

edited Jul 9, 2012 at 2:05

answered Jul 9, 2012 at 1:44

pinkdawn

1,03311 silver badges21 bronze badges

2 Comments

from __future__ Over a year ago

thank you. I'm complaining why I can accept two nice answers:(

pinkdawn Over a year ago

:P you cannot, @from__future__ , but you can still mark this one as "useful"

Collectives™ on Stack Overflow

threading, multiprocessing error

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related