How to properly implement a process based lock in Python?

Question

Problem statement:

I have a script that is called from an external tool and maintains a state over its instances in a set of files. I think that the most practical way to deal with it is to simply serialize the script instances using a single lock with the capability of being acquired (a) when it has not yet been locked, (b) after it has been released and (c) after the process holding it disappeared.

I'm not yet certain whether it is necessary to wake up the next waiting process immediately when an existing process crashes or something, as that is an exceptional condition anyway. But certainly the next action (possibly triggered by a restart) must be able to successfully run.

The script depends on NetworkManager which in turn currently only runs on Linux. Therefore a simple solution is preferred over a cross platform one. On the other hand, a cross platform solution may be useful to a larger number of stackoverflow visitors.

Further discussion:

I found a number of related questions and answers here on stackoverflow but (1) the questions were not as specific as this one and (2) the answers didn't seem to be usable for this case. Especially the part about handling stale locks went mostly unaddressed.

I would like to keep using the context manager API and only libraries common in Linux installation. I guess there's no perfect solution in the standard library nor in any of the common installations, so I think I will need to implement the context manager using some lower level API.

The current code uses the lockfile module which doesn't seem to care about stale locks at all. The script instances aren't expected to share anything except the file system, therefore multiprocessing module based solutions don't seem to apply here. I was thinking about a combination of pidfile and fcntl, but also about a unix socket that could be used for waiting on the other script to finish. I wonder why I can't find a standard context manager based tool for this in Python.

A live version (will change as new patches are accepted) of the script in question:

http://www.nlnetlabs.nl/svn/dnssec-trigger/trunk/dnssec-trigger-script.in

Relevant part of the source code:

def run(self):
    with lockfile.FileLock("/var/run/dnssec-trigger/dnssec-trigger"):
        log.debug("Running: {}".format(self.method.__name__))
        self.method()

The problem is basically described in the first paragraph and the question is what's the best solution to the problem. — Pavel Šimerda
– Pavel Šimerda, Commented Jul 17, 2014 at 9:48

Thomas Orozco · Accepted Answer · 2014-07-18 07:15:46Z

2

You can create your own context manager with contextlib, and use fcntl to issue the locking calls. Note that these can be made non-blocking.

Both contextlib and fcntl are part of the standard library.

For you to experiment with stale locks, you can try launching the process twice and issuing a SIGKILL to one of the two — you should see the lock being released on the other process.

import fcntl
import contextlib

@contextlib.contextmanager
def lock(fname):
    with open(fname, "w") as f:
        print "Acquiring lock"
        fcntl.lockf(f, fcntl.LOCK_EX)
        print "Acquired lock"

        yield

        print "Releasing lock"
        fcntl.lockf(f, fcntl.LOCK_UN)
        print "Released lock"


if __name__ == "__main__":
    import os
    print "PID:", os.getpid()

    import time
    print "Starting"
    with lock("/tmp/lock-file"):
        time.sleep(100)
    print "Done"

edited Jul 18, 2014 at 7:15

answered Jul 17, 2014 at 8:03

Thomas Orozco

55.6k12 gold badges120 silver badges120 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Pavel Šimerda Over a year ago

Does it address the stale locks? Why do you claim that the file has to exist already? Can a directory be used instead of an ordinary file?

Pavel Šimerda Over a year ago

So it seems the stale locks are taken care of (by a simple test with interactive python shells and ctrl+C). I'm curious whether the contextlib.contextmanager is good for anything else than code obfuscation, what about... fpaste.org/118683/55912351 ?

Thomas Orozco Over a year ago

@PavelŠimerda The code presented here and your version are functionally equivalent (you might want to inherit from object though, and use self.lock instead of f). I think that generally speaking contextlib.contextmanager is a staple of context managers, so most Python developers should be familiar with it, but that doesn't mean you have to use it.

Pavel Šimerda Over a year ago

Thanks for the notes on bugs in the implementation. I still wonder why one would use a generator instead of an explicit class. Inheriting object should be automatic in Python 3 and I'm not sure whether Python 2 makes a big difference here. Regarding the need to create the file, I think you can rely on open() to do that and therefore you can simply remove the comment from the answer.

Pavel Šimerda Over a year ago

While I like the class based code better, I'll accept this answer as it does what was asked for and my answer is based on this one anyway.

Pavel Šimerda · Accepted Answer · 2014-07-29 10:46:00Z

I submitted the following implementation upstream:

class Lock:
    """Lock used to serialize the script"""

    path = "/var/run/dnssec-trigger/lock"

    def __init__(self):
        # We don't use os.makedirs(..., exist_ok=True) to ensure Python 2 compatibility
        dirname = os.path.dirname(self.path)
        if not os.path.exists(dirname):
            os.makedirs(dirname)
        self.lock = open(self.path, "w")

    def __enter__(self):
        fcntl.lockf(self.lock, fcntl.LOCK_EX)

    def __exit__(self, t, v, tb):
        fcntl.lockf(self.lock, fcntl.LOCK_UN)

It's not entirely generic (path name is hardcoded), it doesn't close the file descriptor and it may possibly be improved in other ways but I still want to include it for reference.

Collectives™ on Stack Overflow

How to properly implement a process based lock in Python?

2 Answers 2

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related