Error when mkdir in multi threads in python

Question

I have code in multi threads to create folder if not exists

if not os.path.exists(folder): os.makedirs(folder)

I got error like this

The folder cannot be created since a file already exists with the same path

I am not sure what can I do for this error, do you have any idea?

Do you have multiple threads trying to create the same folder? The problem is that thread 1 might create the folder between the time thread 2 checks if it exists and then tries to create it. — Barmar
– Barmar, Commented Mar 2, 2017 at 0:46
you need to implement mutual exclusion between the threads, so that the test and creation are performed atomically. — Barmar
– Barmar, Commented Mar 2, 2017 at 0:47
I am afraid performance if I use Mutual. In fact, I check and create folder in main thread at first, but I don't want to create folder if no any image download so that I move that function to just before save in threads, is there any way to not affect performance and do what I need? — mikezang
– mikezang, Commented Mar 2, 2017 at 1:07
Unless you're calling this code frequently, don't worry about the performance. Get it right first, then worry about performance if it's a bottleneck. — Barmar
– Barmar, Commented Mar 2, 2017 at 1:10
You could just try to create the directory, and ignore the error if it already exists. Then whichever thread gets here first will create it. — Barmar
– Barmar, Commented Mar 2, 2017 at 1:11

ShadowRanger · Accepted Answer · 2017-03-02 01:39:01Z

15

Read the docs. If you don't care whether the directory already existed, just that it does when you're done, just call:

os.makedirs(folder, exist_ok=True)

Don't even check for the existence of the directory with exists (subject to race conditions), just call os.makedirs with exist_ok=True and it will create it if it doesn't exist and do nothing if it already exists.

This requires Python 3.2 or higher, but if you're on an earlier Python, you can achieve the same silent ignore with exception handling:

import errno

try:
    os.makedirs(folder)
except OSError as e:
    if e.errno != errno.EEXIST:
        raise  # Reraise if failed for reasons other than existing already

answered Mar 2, 2017 at 1:39

ShadowRanger

158k12 gold badges222 silver badges317 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

mikezang Over a year ago

Great! This is what I need!

M. Bauer Over a year ago

I just want to add that this is also true for pathlib.Path: dest.mkdir(parents=True, exist_ok=True)

Leedehai Over a year ago

It appears to not be the case. bugs.python.org/issue21082#msg215346 points out exist_ok is not thread-safe.

Antonio Bri Over a year ago

Do you know if it is already fixed? @Leedehai

ShadowRanger Over a year ago

@AntonioBri: The race Leedehai refers to appears to only be possible if multiple threads are both creating and removing/renaming the directory in question. As long as you're only creating the directory, the race doesn't occur. The code has changed since then, but only insofar as it simplified the check on OSError to just if not exist_ok or not path.isdir(name): raise, so if multiple threads are both creating the directory and removing/renaming it, then a race remains possible.

Collectives™ on Stack Overflow

Error when mkdir in multi threads in python

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related