3

What is the best practice to try to open the file and retry after n seconds?

Currently, I do:

import os
from os import path
import shutil

dir_path = path.abspath(path.join("foo", "bar"))
destination_path = path.abspath(path.join("foo", "dest_dir"))

for f in dir_path:
    try:
        # try to open the file if the file isn't used any longer
        file_opened = open(f, 'r')
        # move the file on successful opening
        shutil.move(file_opened, destination_path)
        file_opened.close()
    except IOError:
        return False

So, at the moment I do not handle the exception. I think about creation of extra function to open the file and recall the function on excepth with time.sleep(n)

However, I am sure there must be something else ...

I do not use 

with open(f, 'rb') as file_opened: 
    do whatever`

EDIT:

One process creates the file and I want Python process to move the file once I am sure the file writting / creation is completed. So, I have added shutil.move in the above code to show the whole situation.

EDIT:

Please find below the code I have developed to solve the problem. I ended with writing own custom solution to handle it:

import os
from os import path
import psutil
from retry import retry
import shutil
from subprocess import check_output,Popen, PIPE
import glob
import time


class FileHandler:
    def __init__(self, fn_source, dir_source):
        self.file_source_name = fn_source
        self.file_source_path = path.join(dir_source, self.file_source_name)
        self.dir_dest_path = path.join(dir_source, "test")
        self.file_dest_path = path.join(self.dir_dest_path, self.file_source_name)

    def check_file(self):
        if os.path.exists(self.file_source_path):
            try:
                os.rename(self.file_source_path, self.file_source_path)
                print("file renamed")
                return True
            except:
                print("can not rename the file..retrying")
                time.sleep(1)
                self.check_file()
        else:
            print("source file does not exist...retrying")
            time.sleep(5)
            self.check_file()

    def check_destination(self):
        if os.path.exists(self.file_source_path) and not os.path.exists(self.file_dest_path):
            return True
        elif os.path.exists(self.file_source_path) and os.path.exists(self.file_dest_path):
            try:
                print(self.file_dest_path, self.file_source_name)
                os.remove(self.file_dest_path)
                return True
            except Exception as e:
                print("can not remove the file..retrying")
                time.sleep(5)
                self.check_destination()

    def move_file(self):
        if self.check_destination():
            print(self.file_source_path)
            shutil.move(self.file_source_path, self.file_dest_path)
            print("moved", str(self.file_source_path))
            return True
        else:
            print("can not move the file..retrying")
            time.sleep(1)
            self.move_file()

    def file_ops(self):
        if self.check_file():
            self.move_file()
        else:
            print("source does not exist")
            time.sleep(1)
            self.file_ops()
        return True


def each_file_ops(fn, dir_source):
    fh = FileHandler(fn, dir_source)
    return fh.file_ops()


def main(dir_source):
    dir_files = glob.glob(path.join(dir_source, '*.txt'))
    if dir_files:
        [each_file_ops(f, dir_source) for f in dir_files]
    else:
        print("source dir is empty")
        time.sleep(1)
        main(dir_source)


if __name__ == '__main__':
    main(path.join(""))
Improve this question
10
  • 2
    What is the point of trying to open the file that could not be opened? Commented Jun 21, 2018 at 6:16
  • The point is that if you can not open the file right here right now because sth may be still using it than you may want to come back and check again to see if you can open the file? Commented Jun 21, 2018 at 6:18
  • e.g. monster1 creates the file for you but there is no way to speak with the monster1 so instead you may want to recheck to try to open the file and see if the monster1 is gone and the file is left for you so you can play with it. Commented Jun 21, 2018 at 6:20
  • The you should wait for the lock to become available, rather than keep trying. Commented Jun 21, 2018 at 6:20
  • What is file_path? When do you use the loop variable f? Commented Jun 21, 2018 at 6:21

2 Answers 2

Reset to default
3

You can use the retry module for these kind of retrying. This makes the code to look much cleaner. pip install retry should install the module

from retry import retry
import shutil

@retry((FileNotFoundError, IOError), delay=1, backoff=2, max_delay=10, tries=100)
def attempt_to_move_file(fname, dest_path):
    # Your attempt to move file
    # shutil.move(fname, destination_path)

With the above code, when attempt_to_move_file is invoked it would be retried (upto a max of 100 tries) whenever we hit FileNotFoundError or IOError and the retry happens with a sleep 1, 2, 4, 8, 10, 10, 10 ... seconds between attempts

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

shutil.move() will still move the file irrespective of the file is open or not, however, if any write operation is happening on the file, it will still keep running on the file in the new location.
no, the shutil.move() will not move the file if the file is used by other process.
@mothership please comment I made under my answer.
Just for the record, as @UmeshKaushik has pointed out using shutil.move() would not be the best solution as it can move partially created files.. Atleast thats the behavior I would see on Linux machines. As few others have pointed out, the best solution is to make the other process creating the file to create and acquire a lock on that file till it completes the writing. You should then wait till this lock getting released in your code
@Sunitha ... I can not influence the writter so I can not force it to set lock etc. hence why my approach to deal with it from the reader perspective.
|
0

you do not need to open a file to move it. Because once you have opened a file it is in open state hence you can not move it, Take it like this, when you are playing a video in a player and you try to delete it or cut and paste system won't allow you. 

import shutil

file_name = 'trytry.csv'

shutil.copyfile(file_name, '/Users/umeshkaushik/PycharmProjects/newtry.csv')
shutil.move(file_name, '/Users/umeshkaushik/PycharmProjects/newtry1.csv')

Above code runs fine. Just make sure you are providing correct input and output path.

Improve this answer

11 Comments

If process A creates the file and write lines to it then how do you check with above code if the process A still is not writting to the file so how do you check if the file is not opened or how do you check the file is not used and the writting is completed? The above code will take the snapshot of the file at the specific point in time while actually writting may be completed few seconds after you take the snapshot.
hence why I do try to open the file and then close it then I move it.
Your statements in your code are wrong. I am not creating the file. Other process creates the file. I am just opening it in read mode to see if I can open it. I close it straight after I open it and then I move it once I am sure I can open & close it so it tell me file is not used.
I agree, made a mistake by using write in the answer! Removing that part.
And for the records, multiple write locks are possible on a file.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.