Cannot get arguments in Python Multiprocessing

Question

import re, os
from multiprocessing import Pool

directory = r'F:\data\\' #['a.txt', 'b.txt', 'c.txt', 'd.txt', 'e.txt']
raw_files = os.listdir(directory)

# TARGET FUNCTION
def print_result(raw_files):
    for raw_file in raw_files:
        with open(directory+raw_file, 'r', encoding = 'utf-16') as f: #FileNotFoundError: [Errno 2] No such file or directory: 'F:\\corpus_duplicated\\\\2'
            raw = f.read()
            if re.search('target', raw):
                print(raw)

if __name__ == '__main__':
    print(raw_files[:3]) #['a.txt', 'b.txt', 'c.txt']
    pool = Pool(processes = 4)
    pool.map(print_result, raw_files)
    pool.close()
    pool.join()

I want to directory+raw_file be F:\data\a.txt but it results in F:\data\2 that can be seen in Error msg.

I think I didn't understand about multiprocessing yet, but I cannot know why via searching.

Thank you for help. (Attatched code is reduced one.)

drops · Accepted Answer · 2020-07-25 07:22:56Z

1

The function print_result should be changed, so that it handles only one file. You give map the list with files and it will chunk the items in the list and hand them in separate processes to print_result

import re, os
from multiprocessing import Pool

directory = 'F:/data/'
raw_files = os.listdir(directory)

# TARGET FUNCTION
# input to this function should be a single file
def print_result(raw_file):  
    with open(directory+raw_file, 'r', encoding = 'utf-16') as f:
        raw = f.read()
        if re.search('target', raw):
            print(raw)

if __name__ == '__main__':
    print(raw_files[:3]) #['a.txt', 'b.txt', 'c.txt']
    pool = Pool(processes = 4)
    pool.map(print_result, raw_files)
    pool.close()
    pool.join()

You might have to change the forward-slashes back to back-slashes for directory

answered Jul 25, 2020 at 7:22

drops

1,6141 gold badge15 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

berry2971 Over a year ago

Oh I wasted long time in other lines. I was not thorough. Thank you!

Collectives™ on Stack Overflow

Cannot get arguments in Python Multiprocessing

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related