How to make file creation an atomic operation?

A simple snippet that implements atomic writing using Python tempfile.

with open_atomic('test.txt', 'w') as f:
    f.write("huzza")

or even reading and writing to and from the same file:

with open('test.txt', 'r') as src:
    with open_atomic('test.txt', 'w') as dst:
        for line in src:
            dst.write(line)

using two simple context managers

import os
import tempfile as tmp
from contextlib import contextmanager

@contextmanager
def tempfile(suffix='', dir=None):
    """ Context for temporary file.

    Will find a free temporary filename upon entering
    and will try to delete the file on leaving, even in case of an exception.

    Parameters
    ----------
    suffix : string
        optional file suffix
    dir : string
        optional directory to save temporary file in
    """

    tf = tmp.NamedTemporaryFile(delete=False, suffix=suffix, dir=dir)
    tf.file.close()
    try:
        yield tf.name
    finally:
        try:
            os.remove(tf.name)
        except OSError as e:
            if e.errno == 2:
                pass
            else:
                raise

@contextmanager
def open_atomic(filepath, *args, **kwargs):
    """ Open temporary file object that atomically moves to destination upon
    exiting.

    Allows reading and writing to and from the same filename.

    The file will not be moved to destination in case of an exception.

    Parameters
    ----------
    filepath : string
        the file path to be opened
    fsync : bool
        whether to force write the file to disk
    *args : mixed
        Any valid arguments for :code:`open`
    **kwargs : mixed
        Any valid keyword arguments for :code:`open`
    """
    fsync = kwargs.pop('fsync', False)

    with tempfile(dir=os.path.dirname(os.path.abspath(filepath))) as tmppath:
        with open(tmppath, *args, **kwargs) as file:
            try:
                yield file
            finally:
                if fsync:
                    file.flush()
                    os.fsync(file.fileno())
        os.rename(tmppath, filepath)

Since it is very easy to mess up with the details, I recommend using a tiny library for that. The advantage of a library is that it takes care all these nitty-gritty details, and is being reviewed and improved by a community.

One such library is python-atomicwrites by untitaker which even has proper Windows support:

• https://github.com/untitaker/python-atomicwrites
• https://pypi.org/project/atomicwrites/

Caveat (as of 2023):

This library is curently unmaintained. Comment from the author:

[...], I thought it'd be a good time to deprecate this package. Python 3 has os.replace and os.rename which probably do well enough of a job for most usecases.

Original recommendation:

From the README:

from atomicwrites import atomic_write

with atomic_write('foo.txt', overwrite=True) as f:
    f.write('Hello world.')
    # "foo.txt" doesn't exist yet.

# Now it does.

Installation via PIP:

pip install atomicwrites

I’m using this code to atomically replace/write a file:

import os
from contextlib import contextmanager

@contextmanager
def atomic_write(filepath, binary=False, fsync=False):
    """ Writeable file object that atomically updates a file (using a temporary file).

    :param filepath: the file path to be opened
    :param binary: whether to open the file in a binary mode instead of textual
    :param fsync: whether to force write the file to disk
    """

    tmppath = filepath + '~'
    while os.path.isfile(tmppath):
        tmppath += '~'
    try:
        with open(tmppath, 'wb' if binary else 'w') as file:
            yield file
            if fsync:
                file.flush()
                os.fsync(file.fileno())
        os.rename(tmppath, filepath)
    finally:
        try:
            os.remove(tmppath)
        except (IOError, OSError):
            pass

Usage:

with atomic_write('path/to/file') as f:
    f.write("allons-y!\n")

It’s based on this recipe.

Just link the file after you're done:

with tempfile.NamedTemporaryFile(mode="w") as f:
    f.write(...)
    os.link(f.name, final_filename)

If you want to get fancy:

@contextlib.contextmanager
def open_write_atomic(filename: str, **kwargs):
    kwargs['mode'] = 'w'
    with tempfile.NamedTemporaryFile(**kwargs) as f:
        yield f
        os.link(f.name, filename)

Answers on this page are quite old, there are now libraries that do this for you.

In particular safer is a library designed to help prevent programmer error from corrupting files, socket connections, or generalized streams. It's quite flexible and amongst other things it has the option to use either memory or temporary files, you can even keep the temp files in case of failure.

Their example is just what you want:

# dangerous
with open(filename, 'w') as fp:
    json.dump(data, fp)
    # If an exception is raised, the file is empty or partly written

# safer
with safer.open(filename, 'w') as fp:
    json.dump(data, fp)
    # If an exception is raised, the file is unchanged.

It's in PyPI, just install it using pip install --user safer or get the latest at https://github.com/rec/safer

Atomic solution for Windows to loop folder and rename files. Tested, atomic to automate, you can increase probability to minimize risk not to event of having same file name. You random library for letter symbols combinations use random.choice method, for digit str(random.random.range(50,999999999,2). You can vary digits range as you want.

import os import random

path = "C:\\Users\\ANTRAS\\Desktop\\NUOTRAUKA\\"

def renamefiles():
    files = os.listdir(path)
    i = 1
    for file in files:
        os.rename(os.path.join(path, file), os.path.join(path, 
                  random.choice('ABCDEFGHIJKL') + str(i) + str(random.randrange(31,9999999,2)) + '.jpg'))
        i = i+1

for x in range(30):
    renamefiles()