Locking a file when using multiprocessing and multithreading - Python

Question

I have a main.py file. This file uses multiprocessing to execute another file called function.py. The second one use threading to apply a function, f, to every component of a numpy array. function.py reads (only once in all the process) a file, file.txt, to read some data for f and then to clear it (write a empty file). Do I need to lock the file, file.txt, in function.py to avoid problems of having N processes, created from main.py executing function.py and reading and writing in file.txt? If so, how can it be done?

Finally I get it with a semaphore.

Andreas · Accepted Answer · 2017-07-05 08:52:09Z

1

Yes, in some way it has to be locked. Having several processes reading a file is no problem as long as they are only reading it. As soon as something writes to the file you have to be sure that the reads and writes occur in the desired order.

Locking could be done by using a lockfile that is atomically created. After a process successfully creates the lockfile, it gains access to the text file. After the process is done with the text file, it deletes the lockfile. This assures that only one process can access the text file at a given time.

answered Jul 5, 2017 at 8:52

Andreas

948 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Andreas Over a year ago

You can also synchronize your process inside the program to make sure only one process at a time accesses the code where the file is edited.

Ishan Srivastava Over a year ago

best answer to the question

Collectives™ on Stack Overflow

Locking a file when using multiprocessing and multithreading - Python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related