405

Run the following code from a directory that contains a directory named bar (containing one or more files) and a directory named baz (also containing one or more files). Make sure there is not a directory named foo.

import shutil
shutil.copytree('bar', 'foo')
shutil.copytree('baz', 'foo')

It will fail with:

$ python copytree_test.py 
Traceback (most recent call last):
  File "copytree_test.py", line 5, in <module>
    shutil.copytree('baz', 'foo')
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/shutil.py", line 110, in copytree
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/os.py", line 172, in makedirs
OSError: [Errno 17] File exists: 'foo'

I want this to work the same way as if I had typed:

$ mkdir foo
$ cp bar/* foo/
$ cp baz/* foo/

Do I need to use shutil.copy() to copy each file in baz into foo? (After I've already copied the contents of 'bar' into 'foo' with shutil.copytree()?) Or is there an easier/better way?

Improve this question
3

16 Answers 16

Reset to default
479

Here's a solution that's part of the standard library:

For Python 3.7 and earlier:

from distutils.dir_util import copy_tree
copy_tree("/a/b/c", "/x/y/z")

For Python 3.8+, they added the dirs_exist_ok parameter:

import shutil
shutil.copytree("/a/b/c", "/x/y/z", dirs_exist_ok=True)
Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

This is a good one because it uses the standard library. Symlinks, mode and time can be preserved as well.
Noticed a small disadvantage. distutils.errors.DistutilsInternalError: mkpath: 'name' must be a string, i.e. it does not accept PosixPath. Need to str(PosixPath). Wish list for improvement. Other than this matter, I prefer this answer.
@SunBear, Yeah, I think that's going to be the case with most of the other libraries that take paths as strings. Part of the downside to choosing not to make the Path object inherit from str I suppose, like most of the prior implementations of object oriented path objects..
While "technically public", please note that the developers of distutils made it clear (same link as @SunBear's, thx!) that distutils.dir_util.copy_tree() is considered an implementation detail of distutils and not recommended for public use. The real solution should be for shutil.copytree() to be improved/extended to behave more like distutils.dir_util.copy_tree(), but without its shortcomings. In the meantime, I'll keep using custom helper functions similar to some of those provided in other answers.
distutils has been deprecated in Python 3.10 and is planned to be removed in 3.12, see PEP 632 for info.
|
263

This limitation of the standard shutil.copytree seems arbitrary and annoying. Workaround:

import os, shutil
def copytree(src, dst, symlinks=False, ignore=None):
    for item in os.listdir(src):
        s = os.path.join(src, item)
        d = os.path.join(dst, item)
        if os.path.isdir(s):
            shutil.copytree(s, d, symlinks, ignore)
        else:
            shutil.copy2(s, d)

Note that it's not entirely consistent with the standard copytree:

  • it doesn't honor symlinks and ignore parameters for the root directory of the src tree;
  • it doesn't raise shutil.Error for errors at the root level of src;
  • in case of errors during copying of a subtree, it will raise shutil.Error for that subtree instead of trying to copy other subtrees and raising single combined shutil.Error.
Improve this answer

13 Comments

Thanks! Agree that this seems totally arbitrary! shutil.copytree does an os.makedirs(dst) at the start. No part of the code actually would have a problem with a pre-existing dir. This needs to be changed. At least provide an exist_ok=False parameter to the call
This is a good answer - however Mital Vora's answer below is worth looking at also. They have called copytree recursively rather than calling shutil.copytree() because the same problem will arise otherwise. Possibly consider merging answers or updating to Mital Vora's.
This fails if given a path that includes a directory which is not empty in the destination. Maybe somebody could solve this with tail recursion but here's a modification to your code that works def copyTree( src, dst, symlinks=False, ignore=None): for item in os.listdir(src): s = os.path.join(src, item) d = os.path.join(dst, item) if os.path.isdir(s): if os.path.isdir(d): self.recursiveCopyTree(s, d, symlinks, ignore) else: shutil.copytree(s, d, symlinks, ignore) else: shutil.copy2(s, d)
Meh, super-annoying. It's 4 years later, and shutil.copytree still has this silly restriction. :-(
@antred ...but distutils.dir_util.copy_tree(), which also resides in the stdlib, has no such restriction and actually behaves as expected. Given that, there's no compelling reason to attempt to unroll your own (...typically broken) implementation. Brendan Abel's answer should absolutely be the accepted solution now.
|
177

Python 3.8 introduced the dirs_exist_ok argument to shutil.copytree:

Recursively copy an entire directory tree rooted at src to a directory named dst and return the destination directory. dirs_exist_ok dictates whether to raise an exception in case dst or any missing parent directory already exists.

Therefore, with Python 3.8+ this should work:

import shutil

shutil.copytree('bar', 'foo')  # Will fail if `foo` exists
shutil.copytree('baz', 'foo', dirs_exist_ok=True)  # Fine
Improve this answer

4 Comments

@Jay, only if the directory already exists. I left dirs_exist_ok out of the first call to illustrate the difference (and because the directory doesn't yet exist in OP's example), but of course you can use it if you want.
this also works with pathlib.Path objects as src and dst arguments :) distutils.dir_util's copy_tree on the other hand requires conversion to string.
This indeed correctly copies the contents of bar into foo, as OP suggested. Is it possible to copy the whole bar dir into foo so that you end up with a structure like bar/foo?
@pcko1, assuming you mean foo/bar/ and not bar/foo/, try shutil.copytree("bar", "foo/bar", dirs_exist_ok=True). A pathlib.Path object works as well in case you don't want to hard-code the / directory separator.
75

In slight improvement on atzz's answer to the function where the above function always tries to copy the files from source to destination. 

def copytree(src, dst, symlinks=False, ignore=None):
    if not os.path.exists(dst):
        os.makedirs(dst)
    for item in os.listdir(src):
        s = os.path.join(src, item)
        d = os.path.join(dst, item)
        if os.path.isdir(s):
            copytree(s, d, symlinks, ignore)
        else:
            if not os.path.exists(d) or os.stat(s).st_mtime - os.stat(d).st_mtime > 1:
                shutil.copy2(s, d)

In my above implementation

  • Creating the output directory if not already exists
  • Doing the copy directory by recursively calling my own method.
  • When we come to actually copying the file I check if the file is modified then only we should copy.

I am using above function along with scons build. It helped me a lot as every time when I compile I may not need to copy entire set of files.. but only the files which are modified.

Improve this answer

6 Comments

Nice, except that you have symlinks and ignore as arguments, but they are ignored.
It is worth noting that st_mtime granularity can be as coarse as 2 seconds on FAT filesystems docs.python.org/2/library/os.html. Using this code in a context where updates happen in rapid succession, you may find overrides don't take place.
There is a bug in the second-to-last line, should be: if not os.path.exists(d) or os.stat(s).st_mtime - os.stat(d).st_mtime > 1:
What's the purpose of the third and fourth parameters of copytree()? Those 2 parameters -- symlinks, ignore -- are never used, so they could be ommitted, right?
pls check shutil.copytree documentation docs.python.org/3/library/shutil.html for more details on ignore parameter.
|
41

A merge one inspired by atzz and Mital Vora:

#!/usr/bin/python
import os
import shutil
import stat
def copytree(src, dst, symlinks = False, ignore = None):
  if not os.path.exists(dst):
    os.makedirs(dst)
    shutil.copystat(src, dst)
  lst = os.listdir(src)
  if ignore:
    excl = ignore(src, lst)
    lst = [x for x in lst if x not in excl]
  for item in lst:
    s = os.path.join(src, item)
    d = os.path.join(dst, item)
    if symlinks and os.path.islink(s):
      if os.path.lexists(d):
        os.remove(d)
      os.symlink(os.readlink(s), d)
      try:
        st = os.lstat(s)
        mode = stat.S_IMODE(st.st_mode)
        os.lchmod(d, mode)
      except:
        pass # lchmod not available
    elif os.path.isdir(s):
      copytree(s, d, symlinks, ignore)
    else:
      shutil.copy2(s, d)
  • Same behavior as shutil.copytree, with symlinks and ignore parameters
  • Create directory destination structure if non existant
  • Will not fail if dst already exists
Improve this answer

7 Comments

This is much much faster than original solution when the directory nesting is deep. Thanks
Did you define a function also named 'ignore' in code elsewhere?
You can define any function with any name you like before calling copytree function. This function (which could also be a lambda expression) takes two arguments: a directory name and the files in it, it should return an iterable of ignore files.
[x for x in lst if x not in excl] this does not do the same as copytree, which uses glob pattern matching. en.wikipedia.org/wiki/Glob_(programming)
This is great. The ignore was not being utilized correctly in the above answer.
|
8

docs explicitly state that destination directory should not exist:

The destination directory, named by dst, must not already exist; it will be created as well as missing parent directories.

I think your best bet is to os.walk the second and all consequent directories, copy2 directory and files and do additional copystat for directories. After all that's precisely what copytree does as explained in the docs. Or you could copy and copystat each directory/file and os.listdir instead of os.walk.

Improve this answer

Comments

1

This is inspired from the original best answer provided by atzz, I just added replace file / folder logic. So it doesn't actually merge, but deletes the existing file/ folder and copies the new one:

import shutil
import os
def copytree(src, dst, symlinks=False, ignore=None):
    for item in os.listdir(src):
        s = os.path.join(src, item)
        d = os.path.join(dst, item)
        if os.path.exists(d):
            try:
                shutil.rmtree(d)
            except Exception as e:
                print e
                os.unlink(d)
        if os.path.isdir(s):
            shutil.copytree(s, d, symlinks, ignore)
        else:
            shutil.copy2(s, d)
    #shutil.rmtree(src)

Uncomment the rmtree to make it a move function.

Improve this answer

Comments

1

Here is my pass at the problem. I modified the source code for copytree to keep the original functionality, but now no error occurs when the directory already exists. I also changed it so it doesn't overwrite existing files but rather keeps both copies, one with a modified name, since this was important for my application. 

import shutil
import os


def _copytree(src, dst, symlinks=False, ignore=None):
    """
    This is an improved version of shutil.copytree which allows writing to
    existing folders and does not overwrite existing files but instead appends
    a ~1 to the file name and adds it to the destination path.
    """

    names = os.listdir(src)
    if ignore is not None:
        ignored_names = ignore(src, names)
    else:
        ignored_names = set()

    if not os.path.exists(dst):
        os.makedirs(dst)
        shutil.copystat(src, dst)
    errors = []
    for name in names:
        if name in ignored_names:
            continue
        srcname = os.path.join(src, name)
        dstname = os.path.join(dst, name)
        i = 1
        while os.path.exists(dstname) and not os.path.isdir(dstname):
            parts = name.split('.')
            file_name = ''
            file_extension = parts[-1]
            # make a new file name inserting ~1 between name and extension
            for j in range(len(parts)-1):
                file_name += parts[j]
                if j < len(parts)-2:
                    file_name += '.'
            suffix = file_name + '~' + str(i) + '.' + file_extension
            dstname = os.path.join(dst, suffix)
            i+=1
        try:
            if symlinks and os.path.islink(srcname):
                linkto = os.readlink(srcname)
                os.symlink(linkto, dstname)
            elif os.path.isdir(srcname):
                _copytree(srcname, dstname, symlinks, ignore)
            else:
                shutil.copy2(srcname, dstname)
        except (IOError, os.error) as why:
            errors.append((srcname, dstname, str(why)))
        # catch the Error from the recursive copytree so that we can
        # continue with other files
        except BaseException as err:
            errors.extend(err.args[0])
    try:
        shutil.copystat(src, dst)
    except WindowsError:
        # can't copy file access times on Windows
        pass
    except OSError as why:
        errors.extend((src, dst, str(why)))
    if errors:
        raise BaseException(errors)
Improve this answer

3 Comments

Recursive nature of this function is potentially problematic in python which is not made for deep recursive calls. Depends on depth of directories and if it's a performance problem I guess.
This is technically true, although in practice I think you are unlikely to encounter such a case. The max recursion depth in python is 1000 by default, so you would need to have a directory structure 1000 folders deep. (Note you can have as many directories as you want in the first level, and for Python's purposes all those calls would count as recursion depth 1). Filepaths are limited to 4k characters so it might actually be rather challenging to make a meaningful file structure 1000 folders deep being limited to three character names or shorter.
This might not happen much in reality and it is limited to 4k for some system functions, but many common filesystems have no filepath and with that recursive subdirectory limit. See askubuntu.com/questions/859945/… and en.wikipedia.org/wiki/Comparison_of_file_systems#Limits
1

Here is a version that expects a pathlib.Path as input.

# Recusively copies the content of the directory src to the directory dst.
# If dst doesn't exist, it is created, together with all missing parent directories.
# If a file from src already exists in dst, the file in dst is overwritten.
# Files already existing in dst which don't exist in src are preserved.
# Symlinks inside src are copied as symlinks, they are not resolved before copying.
#
def copy_dir(src, dst):
    dst.mkdir(parents=True, exist_ok=True)
    for item in os.listdir(src):
        s = src / item
        d = dst / item
        if s.is_dir():
            copy_dir(s, d)
        else:
            shutil.copy2(str(s), str(d))

Note that this function requires Python 3.6, which is the first version of Python where os.listdir() supports path-like objects as input. If you need to support earlier versions of Python, you can replace listdir(src) by listdir(str(src)).

Improve this answer

1 Comment

Since I couldn't edit, I have modified version of @Boris-Dalstein code and added as an anser
0

Here is my version of the same task::

import os, glob, shutil

def make_dir(path):
    if not os.path.isdir(path):
        os.mkdir(path)


def copy_dir(source_item, destination_item):
    if os.path.isdir(source_item):
        make_dir(destination_item)
        sub_items = glob.glob(source_item + '/*')
        for sub_item in sub_items:
            copy_dir(sub_item, destination_item + '/' + sub_item.split('/')[-1])
    else:
        shutil.copy(source_item, destination_item)
Improve this answer

1 Comment

Recursive nature of this function is potentially problematic in python which is not made for deep recursive calls. Depends on depth of directories and if it's a performance problem I guess.
0

Here is a version inspired by this thread that more closely mimics distutils.file_util.copy_file.

updateonly is a bool if True, will only copy files with modified dates newer than existing files in dst unless listed in forceupdate which will copy regardless.

ignore and forceupdate expect lists of filenames or folder/filenames relative to src and accept Unix-style wildcards similar to glob or fnmatch.

The function returns a list of files copied (or would be copied if dryrun if True).

import os
import shutil
import fnmatch
import stat
import itertools

def copyToDir(src, dst, updateonly=True, symlinks=True, ignore=None, forceupdate=None, dryrun=False):

    def copySymLink(srclink, destlink):
        if os.path.lexists(destlink):
            os.remove(destlink)
        os.symlink(os.readlink(srclink), destlink)
        try:
            st = os.lstat(srclink)
            mode = stat.S_IMODE(st.st_mode)
            os.lchmod(destlink, mode)
        except OSError:
            pass  # lchmod not available
    fc = []
    if not os.path.exists(dst) and not dryrun:
        os.makedirs(dst)
        shutil.copystat(src, dst)
    if ignore is not None:
        ignorepatterns = [os.path.join(src, *x.split('/')) for x in ignore]
    else:
        ignorepatterns = []
    if forceupdate is not None:
        forceupdatepatterns = [os.path.join(src, *x.split('/')) for x in forceupdate]
    else:
        forceupdatepatterns = []
    srclen = len(src)
    for root, dirs, files in os.walk(src):
        fullsrcfiles = [os.path.join(root, x) for x in files]
        t = root[srclen+1:]
        dstroot = os.path.join(dst, t)
        fulldstfiles = [os.path.join(dstroot, x) for x in files]
        excludefiles = list(itertools.chain.from_iterable([fnmatch.filter(fullsrcfiles, pattern) for pattern in ignorepatterns]))
        forceupdatefiles = list(itertools.chain.from_iterable([fnmatch.filter(fullsrcfiles, pattern) for pattern in forceupdatepatterns]))
        for directory in dirs:
            fullsrcdir = os.path.join(src, directory)
            fulldstdir = os.path.join(dstroot, directory)
            if os.path.islink(fullsrcdir):
                if symlinks and dryrun is False:
                    copySymLink(fullsrcdir, fulldstdir)
            else:
                if not os.path.exists(directory) and dryrun is False:
                    os.makedirs(os.path.join(dst, dir))
                    shutil.copystat(src, dst)
        for s,d in zip(fullsrcfiles, fulldstfiles):
            if s not in excludefiles:
                if updateonly:
                    go = False
                    if os.path.isfile(d):
                        srcdate = os.stat(s).st_mtime
                        dstdate = os.stat(d).st_mtime
                        if srcdate > dstdate:
                            go = True
                    else:
                        go = True
                    if s in forceupdatefiles:
                        go = True
                    if go is True:
                        fc.append(d)
                        if not dryrun:
                            if os.path.islink(s) and symlinks is True:
                                copySymLink(s, d)
                            else:
                                shutil.copy2(s, d)
                else:
                    fc.append(d)
                    if not dryrun:
                        if os.path.islink(s) and symlinks is True:
                            copySymLink(s, d)
                        else:
                            shutil.copy2(s, d)
    return fc
Improve this answer

Comments

0

The previous solution has some issue that src may overwrite dst without any notification or exception.

I add a predict_error method to predict errors before copy.copytree mainly base on Cyrille Pontvieux's version.

Using predict_error to predict all errors at first is best, unless you like to see exception raised one by another when execute copytree until fix all error.

def predict_error(src, dst):  
    if os.path.exists(dst):
        src_isdir = os.path.isdir(src)
        dst_isdir = os.path.isdir(dst)
        if src_isdir and dst_isdir:
            pass
        elif src_isdir and not dst_isdir:
            yield {dst:'src is dir but dst is file.'}
        elif not src_isdir and dst_isdir:
            yield {dst:'src is file but dst is dir.'}
        else:
            yield {dst:'already exists a file with same name in dst'}

    if os.path.isdir(src):
        for item in os.listdir(src):
            s = os.path.join(src, item)
            d = os.path.join(dst, item)
            for e in predict_error(s, d):
                yield e


def copytree(src, dst, symlinks=False, ignore=None, overwrite=False):
    '''
    would overwrite if src and dst are both file
    but would not use folder overwrite file, or viceverse
    '''
    if not overwrite:
        errors = list(predict_error(src, dst))
        if errors:
            raise Exception('copy would overwrite some file, error detail:%s' % errors)

    if not os.path.exists(dst):
        os.makedirs(dst)
        shutil.copystat(src, dst)
    lst = os.listdir(src)
    if ignore:
        excl = ignore(src, lst)
        lst = [x for x in lst if x not in excl]
    for item in lst:
        s = os.path.join(src, item)
        d = os.path.join(dst, item)
        if symlinks and os.path.islink(s):
            if os.path.lexists(d):
                os.remove(d)
            os.symlink(os.readlink(s), d)
            try:
                st = os.lstat(s)
                mode = stat.S_IMODE(st.st_mode)
                os.lchmod(d, mode)
            except:
                pass  # lchmod not available
        elif os.path.isdir(s):
            copytree(s, d, symlinks, ignore)
        else:
            if not overwrite:
                if os.path.exists(d):
                    continue
            shutil.copy2(s, d)
Improve this answer

1 Comment

Recursive nature of this function is potentially problematic in python which is not made for deep recursive calls. Depends on depth of directories and if it's a performance problem I guess.
0

Try This:

import os,shutil

def copydir(src, dst):
  h = os.getcwd()
  src = r"{}".format(src)
  if not os.path.isdir(dst):
     print("\n[!] No Such directory: ["+dst+"] !!!")
     exit(1)

  if not os.path.isdir(src):
     print("\n[!] No Such directory: ["+src+"] !!!")
     exit(1)
  if "\\" in src:
     c = "\\"
     tsrc = src.split("\\")[-1:][0]
  else:
    c = "/"
    tsrc = src.split("/")[-1:][0]

  os.chdir(dst)
  if os.path.isdir(tsrc):
    print("\n[!] The Directory Is already exists !!!")
    exit(1)
  try:
    os.mkdir(tsrc)
  except WindowsError:
    print("\n[!] Error: In[ {} ]\nPlease Check Your Dirctory Path !!!".format(src))
    exit(1)
  os.chdir(h)
  files = []
  for i in os.listdir(src):
    files.append(src+c+i)
  if len(files) > 0:
    for i in files:
        if not os.path.isdir(i):
            shutil.copy2(i, dst+c+tsrc)

  print("\n[*] Done ! :)")

copydir("c:\folder1", "c:\folder2")
Improve this answer

Comments

0

I couldn't edit the "Boris Dalstein" answer above so here is the improved version of this code:

EDIT on the improvements made:

  • The input args could be str path or pathlib.Path object. Type hint will help.
  • If the source is a directory, it will create that directory as well
  • types are defined for local variables so no warning by the IDE
# Recusively copies the content of the directory src to the directory dst.
# If dst doesn't exist, it is created, together with all missing parent directories.
# If a file from src already exists in dst, the file in dst is overwritten.
# Files already existing in dst which don't exist in src are preserved.
# Symlinks inside src are copied as symlinks, they are not resolved before copying.
#

def copy_dir(source: Union[str, pathlib.Path], destination: Union[str, pathlib.Path]):
    destination_path: pathlib.Path

    if isinstance(source, str):
        source_path = pathlib.Path(source)
    elif isinstance(source, pathlib.Path):
        source_path = source

    if isinstance(destination, str):
        destination_path = pathlib.Path(destination)
    elif isinstance(destination, pathlib.Path):
        destination_path = destination

    destination_path.mkdir(parents=True, exist_ok=True)
    if source_path.is_dir():
        destination_path = destination_path.joinpath(source_path.name)
        destination_path.mkdir(parents=True, exist_ok=True)

    for item in os.listdir(source_path):
        s: pathlib.Path = source_path / item
        d: pathlib.Path = destination_path / item
        if s.is_dir():
            copy_dir(s, d)
        else:
            shutil.copy2(str(s), str(d))
Improve this answer

2 Comments

There are already 18 other answers here. Specifically, how is this better? Why should we use this instead of Boris' answer?
Recursive nature of this function is potentially problematic in python which is not made for deep recursive calls. Depends on depth of directories and if it's a performance problem I guess.
0

You can try this simple fix:

import shutil
shutil.copytree('bar', 'foo')
shutil.copytree('baz', 'foo', dirs_exist_ok=True)

From the Documentation:

If dirs_exist_ok is true, the copying operation will continue if it encounters existing directories, and files within the dst tree will be overwritten by corresponding files from the src tree.

Improve this answer

Comments

-6

i would assume fastest and simplest way would be have python call the system commands...

example..

import os
cmd = '<command line call>'
os.system(cmd)

Tar and gzip up the directory.... unzip and untar the directory in the desired place.

yah?

Improve this answer

4 Comments

if you are running in windows... download 7zip.. and use command line for that. ... again just suggestions.
System commands should always be a last resort. It's always better to utilize the standard library whenever possible so that your code is portable.
Not system independent, requires extra space and CPU time for compression and extracting, definitely not a good idea.
Using system commands can be an effective strategy if the system commands are good and if it's one call which does a lot of things (no problem of the overhead of the call itself), but why zip it and unzip it if it's not transferred over the network?

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.