Why is my multiprocessing-enabled script creating only 4 out of 10 folders?

Question

I am trying to modify my script to copy files across using multiprocessing as an exercise for me learn more about multiprocessing in python

my main does this

if __name__ == "__main__":
    #get command line arguments
    cmdlineArgs = getCmdLineArguments()
    #get all the files in folder
    listOfFiles = getFiles(cmdlineArgs.sourceDirectory)
    #create dataframe of files which needs to be copied
    filesDF = createDF(listOfFiles, cmdlineArgs.destDirectory)
    processes = []
    lstOfDates = list(set(filesDF['date'].to_list()))
    lstOfDates.sort()
    # for dt in lstOfDates:
    #     copyFilesAcross([filesDF, [dt]])
    splitListOfDatesForProc = [(lstOfDates[i:i+3]) for i in range(0, len(lstOfDates), 3)]
    for dt in splitListOfDatesForProc:
        p = Process(target=copyFilesAcross, args=([filesDF, dt],))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()

copyFilesAcross does this :

def copyFilesAcross(lst):
    #keep only the date provided as parameter
    df = lst[0]
    dt = lst[1]
    for d in dt:
        df = df[df.date == d]
        print("Processing date " + d + ' for PID: ', os.getpid())
        for index,row in df.iterrows():
            try:
                #print('Making directory ' + row['destination'])
                os.makedirs(row['destination'], exist_ok=True)
                shutil.copy(row['source'], row['destination'])
            except OSError as e:
                print('Failed to copy file ' + row['source'] + ' with error {0}'.format(e) )
            except:
                print("Unexpected error: ", sys.exc_info()[0])

output :

getFiles: Executed ...
getFiles: Creating empty list ...
getFiles: Concatenating files ...
Creating dataframe of files to be copied ...
Creating empty dataframe ...
Populating dataframe ...
Sorting data frame by date ...
Processing date 20180204 for PID:  35033 <- processed
Processing date 20180304 for PID:  35034 <- processed
Processing date 20180811 for PID:  35038 <- processed
Processing date 20180815 for PID:  35041 <- processed
Processing date 20180311 for PID:  35034 <- not processed
Processing date 20180724 for PID:  35034 <- not processed
Processing date 20180222 for PID:  35033 <- not processed
Processing date 20180303 for PID:  35033 <- not processed
Processing date 20180812 for PID:  35038 <- not processed
Processing date 20180813 for PID:  35038 <- not processed

Process finished with exit code 0

Without multiprocessing the script runs fine so I assume the issue is in the last 2 for loops in main, but I am not sure what I am doing wrong.

Aaron · Accepted Answer · 2021-02-05 04:35:34Z

1

This isn't exactly a multiprocessing problem, you just have a bug in your code.

On the first loop iteration in copyFilesAcross, you overwrite df, and throw away every line other than the one that matches the first date in dt. On the next (and all subsequent) iteration of for d in dt:, you try to find a different date which won't exist, and you then overwrite df with an empty dataframe. When you call for index,row in df.iterrows():, there are no rows, so the loop never executes at all.

answered Feb 5, 2021 at 4:35

Aaron

11.2k1 gold badge27 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Why is my multiprocessing-enabled script creating only 4 out of 10 folders?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related